Voice control system

ABSTRACT

An Internet of Thing (IoT) device checks user authentication using a combination of voice, image, and mobile devices.

BACKGROUND

The present invention relates to the Internet of Things (IoT).

SUMMARY

In one aspect, an Internet of Thing (IoT) device includes a body andsensors including a camera and an accelerometer; a processor; and awireless transceiver coupled to the processor.

Implementations/applications of the above aspect may include one or moreof the following. The ability to network embedded devices with limitedCPU, memory and power resources means that IoT finds applications innearly every field. Such systems could be in charge of collectinginformation in settings ranging from natural ecosystems to buildings andfactories, thereby finding applications in fields of environmentalsensing and urban planning. IoT systems could also be responsible forperforming actions, not just sensing things. Intelligent shoppingsystems, for example, could monitor specific users' purchasing habits ina store by tracking their specific mobile phones. These users could thenbe provided with special offers on their favorite products, or evenlocation of items that they need, which their fridge has automaticallyconveyed to the phone. Additional examples of sensing and actuating arereflected in applications that deal with heat, water, electricity andenergy management, as well as cruise-assisting transportation systems.Other applications that the Internet of things can provide is enablingextended home security features and home automation.

Thus, the Internet of things creates an opportunity to measure, collectand analyze an ever-increasing variety of behavioral statistics.Cross-correlation of this data could revolutionize the targetedmarketing of products and services. Big data and the IoT work inconjunction. From a media perspective, data is the key derivative ofdevice interconnectivity, whilst being pivotal in allowing cleareraccuracy in targeting. The Internet of things therefore transforms themedia industry, companies and even governments, opening up a new era ofeconomic growth and competitiveness. The wealth of data generated bythis industry (i.e. big data) will allow practitioners in advertisingand media to gain an elaborate layer on the present targeting mechanismsused by the industry.

Environmental monitoring applications of the IoT typically use sensorsto assist in environmental protection by monitoring air or waterquality, atmospheric or soil conditions, and can even include areas likemonitoring the movements of wildlife and their habitats. Development ofresource constrained devices connected to the Internet also means thatother applications like earthquake or tsunami early-warning systems canalso be used by emergency services to provide more effective aid. IoTdevices in this application typically span a large geographic area andcan also be mobile.

Monitoring and controlling operations of urban and rural infrastructureslike bridges, railway tracks, on- and offshore-wind-farms is a keyapplication of the IoT. The IoT infrastructure can be used formonitoring any events or changes in structural conditions that cancompromise safety and increase risk. It can also be used for schedulingrepair and maintenance activities in an efficient manner, bycoordinating tasks between different service providers and users ofthese facilities. IoT devices can also be used to control criticalinfrastructure like bridges to provide access to ships. Usage of IoTdevices for monitoring and operating infrastructure is likely to improveincident management and emergency response coordination, and quality ofservice, up-times and reduce costs of operation in all infrastructurerelated areas.

Digital control systems to automate process controls, operator tools andservice information systems to optimize plant safety and security arewithin the purview of the IoT, but it also extends itself to assetmanagement via predictive maintenance, statistical evaluation, andmeasurements to maximize reliability. Smart industrial managementsystems can also be integrated with the Smart Grid, thereby enablingreal-time energy optimization. Measurements, automated controls, plantoptimization, health and safety management, and other functions areprovided by a large number of networked sensors.

The term IIoT (Industrial Internet of Things) is often encountered inthe manufacturing industries, referring to the industrial subset of theIoT. IIoT in manufacturing could generate so much business value that itwill eventually lead to the fourth industrial revolution, so theso-called Industry 4.0. It is estimated that in the future, successfulcompanies will be able to increase their revenue through Internet ofthings by creating new business models and improve productivity, exploitanalytics for innovation, and transform workforce. Integration ofsensing and actuation systems, connected to the Internet, is likely tooptimize energy consumption as a whole. IoT devices will be integratedinto all forms of energy consuming devices (switches, power outlets,bulbs, televisions, etc.) and be able to communicate with the utilitysupply company in order to effectively balance power generation andenergy usage. Such devices would also offer the opportunity for users toremotely control their devices, or centrally manage them via a cloudbased interface, and enable advanced functions like scheduling (e.g.,remotely powering on or off heating systems, controlling ovens, changinglighting conditions etc.). Besides home based energy management, the IoTis especially relevant to the Smart Grid since it provides systems togather and act on energy and power-related information in an automatedfashion with the goal to improve the efficiency, reliability, economics,and sustainability of the production and distribution of electricity.Using metering infrastructure (AMI) devices connected to the Internetbackbone, electric utilities can not only collect data from end-userconnections, but also manage other distribution automation devices liketransformers and reclosers.

IoT devices can be used to enable remote health monitoring and emergencynotification systems. These health monitoring devices can range fromblood pressure and heart rate monitors to advanced devices capable ofmonitoring specialized implants, such as pacemakers Fitbit electronicwristbands or advanced hearing aids. Some hospitals have begunimplementing “smart beds” that can detect when they are occupied andwhen a patient is attempting to get up. It can also adjust itself toensure appropriate pressure and support is applied to the patientwithout the manual interaction of nurses. Specialized sensors can alsobe equipped within living spaces to monitor the health and generalwell-being of senior citizens, while also ensuring that proper treatmentis being administered and assisting people regain lost mobility viatherapy as well. Other consumer devices to encourage healthy living,such as, connected scales or wearable heart monitors, are also apossibility with the IoT. More and more end-to-end health monitoring IoTplatforms are coming up for antenatal and chronic patients, helping onemanage health vitals and recurring medication requirements.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an exemplary environment for communicating data froma monitoring device to external computers

FIG. 1B is a perspective view of an exemplary IoT sport device system.

FIG. 1C is an exemplary process supported by the device according to thepresent invention.

FIG. 2A is a block diagram of an electronic circuit for a smart device.

FIG. 2B is a block diagram of a big data system for predicting stressexperienced by a structural unit such as a bridge, a building, or aplane, for example.

FIG. 3 is a flowchart illustrating one operation of the system of FIG.2A-2B in detecting stress on a unit.

FIG. 4 shows an exemplary sports diagnosis and trainer system foraugmented and/or virtual reality.

FIG. 5 shows an exemplary process for augmented and/or virtual realityfor viewers participating in a game.

FIG. 6 shows an exemplary process to identify reasons for sensor datachanges using a gaming process.

FIG. 7 shows an exemplary glove, FIG. 8 shows an exemplary smart band,FIG. 9 shows exemplary smart clothing, FIG. 10 shows exemplary smartballs.

FIG. 11A shows exemplary smart rackets while FIG. 11B shows electronicsin the handle for golf clubs, rackets, or kung fu sticks.

FIG. 12A-12B show exemplary protective gears, while FIG. 12C shows anexemplary process to fabricate mass-customized protective gear.

FIGS. 13A-13C and 14A-14J show exemplary smart IOT appliances.

FIG. 15A shows an exemplary virtual reality camera mounted on a gear,and FIG. 15B shows exemplary augmented reality real-time coaching of aplayer such as a quarterback during fourth down.

FIG. 16A-16C shows exemplary coaching system for skiing, bicycling, andweightlifting/free style exercise, respectively, while FIG. 16D shows akinematic modeling for detecting exercise motion which in turn allowsprecision coaching suggestions.

Similar reference characters denote corresponding features consistentlythroughout the attached drawings.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The reader should appreciate that the present application describesseveral inventions. Rather than separating those inventions intomultiple isolated patent applications, applicants have grouped theseinventions into a single document because their related subject matterlends itself to economies in the application process. But the distinctadvantages and aspects of such inventions should not be conflated. Insome cases, embodiments address all of the deficiencies noted herein,but it should be understood that the inventions are independentlyuseful, and some embodiments address only a subset of such problems oroffer other, unmentioned benefits that will be apparent to those ofskill in the art reviewing the present disclosure. Due to costsconstraints, some inventions disclosed herein may not be presentlyclaimed and may be claimed in later filings, such as continuationapplications or by amending the present claims. Similarly, due to spaceconstraints, neither the Abstract nor the Summary of the Inventionsections of the present document should be taken as containing acomprehensive listing of all such inventions or all aspects of suchinventions.

FIG. 1A illustrates an exemplary environment for communicating data froma monitoring device to external computers. In FIG. 1A, the monitoringdevice used for a sport device 9 includes an interface with a radiotransmitter for forwarding the result of the comparison to a remotedevice. In one example, the monitoring device may include an additionalswitch and user interface. The user interface may be used by the user inorder to trigger transmission of the comparison of the hand or footpattern reference data with the stroke patterns data to the remotedevice. Alternatively, the transmission may occur automatically eachtime the device has been used, or may be triggered by placing the sportdevice in a cradle or base. All parts of the monitoring device may beencapsulated with each other and/or may be integrated into or attachedto the body of the sport device 9. Alternatively, a radio transmittermay be arranged separately from the other parts, for instance, in abattery charger, cradle or base of the sport device 9. In that example,the interface 7 may include contact terminals in the sport device 9,which are connected to the corresponding terminals in the batterycharger for forwarding the result of the comparison via a wiredconnection to the transmitter in the battery charger or may be connectedby induction or short range wireless communications. The radiotransmitter in the battery charger then transmits this comparison resultfurther via the wireless radio connection to the remote device. In FIG.1A, the remote device may be a mobile phone 16, PDA or computer 19,which receives the information directly from the monitoring device via ashort range radio connection, as one example of a transmitter, such as aBluetooth or a Wifi or a Zigbee connection. In one example, the user ofthe remote device may receive information about how thoroughly the sportdevice 9 has been used or the need to provide a replacement sportdevice. FIG. 1A also illustrates an alternate example of a transmitter,using an intermediate receiver 17 and a network 18, such as a cellularradio system. Also in this example, the radio transmitter may be locatedin connection with the sport device 9 or alternatively in connection,with a charger, cradle or base station of the sport device 9. In such anexample, the comparison result may be transmitted via an intermediatereceiver 17 and the network 18 to a remote device 19, 16 located furtheraway than the range of a short range radio system, for example. Theremove device 19, 16 may be any device suitable for receiving thesignals from the network 18 and providing feedback on an output device.The transmission of information via a cellular radio system to theremote device may allow an advertiser provide an advertisement. Forexample, an advertisement may be added to the comparison result usingnetwork elements in the cellular radio system. The user may receive anadvertisement with the comparison result. An advantage with such asolution is that the advertiser may provide revenue offsetting all or aportion of the cost for the transmission of the comparison result fromthe sport device 9 to the remote device 19, 16.

FIG. 1B shows a block diagram of the unit 9 with processor/RAM/ROM 11.The unit 9 includes a motion sensor, a multi-axis accelerometer, and astrain gage 42. The multi-axis accelerometer may be a two-axis orthree-axis accelerometer. Strain gage 21 is mounted in the neck of theracket, and measures force applied to the ball, i.e., force in a zdirection. Acceleration and force data are acquired by themicroprocessor at a data acquisition rate (sampling rate) of from about10 to 50 samples/second, e.g., about 20 samples/second. The accelerationdata is used to infer motion, using an algorithm discussed below; it isnot converted to position data. In this embodiment, because the sensorsand strain gage are not in the head region, the head can be removableand replaceable, e.g., by threaded engagement with the handle (notshown), so that the sport device can continue to be used afterinstrument wear has occurred. Any desired type of removable head orcartridge can be used.

The unit 11 also includes a camera, which can be a 360 degree camera.Alternatively, the camera can be a 3D camera such as the Kinect cameraor the Intel RealSense camera for ease of generating 3D models and fordetecting distance of objects. To reduce image processing load, eachcamera has a high performance GPU to perform local processing, and theprocessed images, sound, and odor data are uploaded to a cloud storagefor subsequent analysis.

The unit 11 includes an electronic nose to detect odor. The electronicnose can simply be a MEMS device acting as a particle counter. Anembodiment of the electronic nose can be used that includes a fanmodule, a gas molecule sensor module, a control unit and an output unit.The fan module is used to pump air actively to the gas molecule sensormodule. The gas molecule sensor module detects the air pumped into bythe fan module. The gas molecule sensor module at least includes a gasmolecule sensor which is covered with a compound. The compound is usedto combine preset gas molecules. The control unit controls the fanmodule to suck air into the electronic nose device. Then the fan moduletransmits an air current to the gas molecule sensor module to generate adetected data. The output unit calculates the detected data to generatea calculation result and outputs an indicating signal to an operator orcompatible host computer according to the calculation result.

An electronic tongue sensor can be provided to sense quality of sweat orliquid. The tongue includes a liquid molecule sensor module, a controlunit and an output unit. Body liquid is applied or swiped on to theliquid molecule sensor module. The molecule sensor module detects theliquid molecules pumped into by the stirring module. The liquid moleculesensor module at least includes a molecule sensor which is covered witha compound. The compound is used to combine preset liquid molecules. Thecontrol unit controls the stirring module to pump liquid to be “tasted”into the electronic tongue device. Then the module transmits a flowcurrent to the liquid molecule sensor module to generate a detecteddata. The output unit calculates the detected data to generate acalculation result and outputs an indicating signal to an operator orcompatible host computer according to the calculation result. Suchelectronic tongue can detect quality of fog or liquid, among others.

In one embodiment for analyzing tooth structure, restorative materialswithin a tooth structure, and disease states of a tooth, the unit 11includes a probe 20 which may be attached to a variety of sport probes,and instruments to afford adaptability to a variety of situations inproviding diagnostic information on an object such as a naturallyoccurring structure, man-made materials placed or found within thestructure, diseased or otherwise affected, infected or effectedstructure, as well as structure that has been eroded, worn by attrition,abraded, abfracted, fractured, crazed, broken or otherwise compromisedthrough sport enthusiast use, misuse, fatigue or longevity of use. Theprobe 20 generates electrical outputs which are interpreted by a smartphone or computer.

In one embodiment, the probe 20 can be a vibratory transducer that sendsout vibrations at known frequency and amplitude. The probe 20 alsoincludes a receiver which can be an accelerometer, for example. Theaccelerometer is attached to the teeth and connected to a computer. Theaccelerometer digitizes the received vibrations and provides them intothe phone or computer. The transducer can be a single piezoelectrictransducer or an array with elements arranged to fit in a mouthpiece oran appliance to be worn over the oral arch. The transducer elements canbe mounted in silicone rubber or other material suitable for dampingmechanical coupling between the elements. Other materials may also beused for the array construction. For example, the transducer may beformed from one or more pieces of piezocomposite material, or anymaterial that converts electrical energy to acoustic energy. Thereceiver can also be positioned to fit in the mouthpiece or appliance.One embodiment of the receiver is an accelerometer, but a suitablepiezoelectric transducer can serve as the receiver as well.

The software in the computer compares these inputs to known vibrationresponses corresponding to striking states on a ball or sport object.The computer 30 displays a response on the computer screen for thatuser.

FIG. 1C schematically shows a method or app 2 which may be implementedby the computing unit 11 shown in FIG. 1B. For example, the app 2 may bea computer implemented method. A computer program may be provided forexecuting the app 2. The app 2 includes code for:

(21) capture user motion with accelerometer or gyroscope

(22) capture VR views through camera and process using GPU

(23) capture user emotion using facial recognition or GSR

(24) model user action using kinematic model

(25) compare user action with idea action

(26) coach user on improvement to user sport techniques.

As shown in FIG. 2A, a microcontroller 155 receives and processessignals from the sensor 112-114, and converts those signals into anappropriate digital electronic format. The microcontroller 155wirelessly transmits tension information in the appropriate digitalelectronic format, which may be encoded or encrypted for securecommunications, corresponding to the sensed traffic and/or crimeindication through a wireless communication module or transceiver 160and antenna 170. Optionally, a camera 140 can be provided to visuallydetect traffic and/or crime and movement of the structure. Whilemonitoring of the smart device 100 traffic and/or crime is continuous,transmission of tension information can be continuous, periodic orevent-driven, such as when the tension enters into a warning oremergency level. Typically the indicated tension enters a warning level,then an emergency level as tension drops below the optimal range, butcorresponding warning and emergency levels above the optimal range canalso be used if supported by the smart device 100. The microcontroller155 is programmed with the appropriate warning and emergency levels, aswell as internal damage diagnostics and self-recovery features.

The tension information can take any form, including a simplewarning/emergency indication that the tension is approaching orexceeding tension specifications, respectively. While under-tension isknown to be the primary cause of structural or mechanical problemsassociated with devices, over-tension can also be a problem and can alsobe reported by the smart device 100.

The sensors can detect force, load, tension and compression forces onthe device such as the device. Other data includes Acceleration;Velocity; Global absolute displacement; Local relative displacement;Rotation; Strain; Stress; Force; and Static-position video. Windspeed/direction; External temperature; weather parameters (rainfall,humidity, solar radiation, etc.); Internal or structural temperature;Mass loading (occupant count, etc.); Static tilt; Fatigue damage;Corrosion; Acoustic emission; and Moving-position video. A force issimply a push or pull to an object and can be detected by a load cell,pressure cell or strain sensor. A Load: Is simply a force applied to astructure. Ex: weight of vehicles or pedestrians, weight of wind pushingon sides. Tension & Compression are internal forces that make a memberlonger or shorter. Tension stretches a member and Compression pushes themember closer together. Acceleration can also be detected byForce-Balance (Servo) Piezoelectric Piezoresistive MEMS. Velocity can bemeasured by force-balance (servo) MEMS, or Mechanical Doppler Heatedwire. A local Displacement sensor can be LVDT/Cable potentiometerAcoustic Optical/laser Temperature Electrical Optical fiber. A rotationsensor can be Gyro MEMS Gyro Tilt Electro-mechanical MEMS. A strainsensor can be a resistance gauge Vibrating wire Optical fiber CorrosionElectrical Chemical sensors. A traffic and/or crime sensor can be amicrophone listening to acoustic emission, or Piezoelectric MEMS, forexample, and sonar sound processing can be used to detect where crimeactivity is coming from.

The sensor 112-114, transceiver 160/antenna 170, and microcontroller 155are powered by and suitable power source, which may optionally includean electromagnetic field (EMF) scavenging device 145, such as thoseknown in the art, that convert ambient EMF (such as that emitted byradio station broadcasts) into small amounts of electrical power. TheEMF scavenging device 145 includes a battery to buffer and store energyfor the microcontroller 155, sensor 112-114, camera 140 and wirelesscommunications 160/170, among others.

The circuit of FIG. 2A contains an analog front-end (“AFE”) transducer150 for interfacing signals from the sensor 112-114 to themicrocontroller 155. The AFE 150 electrically conditions the signalscoming from the sensor 112-114 prior to their conversion by themicrocontroller 155 so that the signals are electrically compatible withthe specified input ranges of the microcontroller 155. Themicrocontroller 155 can have a CPU, memory and peripheral circuitry. Themicrocontroller 155 is electrically coupled to a wireless communicationmodule 160 using either a standard or proprietary communicationstandard. Alternatively, the microcontroller 155 can include internallyany or all circuitry of the smart device 100, including the wirelesscommunication module 160. The microcontroller 155 preferably includespower savings or power management circuitry 145 and modes to reducepower consumption significantly when the microcontroller 155 is notactive or is less active. The microcontroller 155 may contain at leastone Analog-to-Digital Converter (ADC) channel for interfacing to the AFE150.

The battery/power management module 145 preferably includes theelectromagnetic field (EMF) scavenging device, but can alternatively runoff of previously stored electrical power from the battery alone. Thebattery/power management module 145 powers all the circuitry in thesmart device 100, including the camera 140, AFE 150, microcontroller155, wireless communication module 160, and antenna 170. Even though thesmart device 100 is preferably powered by continuously harvesting RFenergy, it is beneficial to minimize power consumption. To minimizepower consumption, the various tasks performed by the circuit should berepeated no more often than necessary under the circumstances.

Stress information from the smart device 100 and other information fromthe microcontroller 155 is preferably transmitted wirelessly through awireless communication module 160 and antenna 170. As stated above, thewireless communication component can use standard or proprietarycommunication protocols. Smart lids 100 can also communicate with eachother to relay information about the current status of the structure ormachine and the smart device 100 themselves. In each smart device 100,the transmission of this information may be scheduled to be transmittedperiodically. The smart lid 100 has a data storage medium (memory) tostore data and internal status information, such as power levels, whilethe communication component is in an OFF state between transmissionperiods. On the other hand, once the communication commences in the ONstate, the microcontroller 155 can execute the following tasks:

1. Neighbor discovery: in this task each smart device 100 sends a beaconidentifying its location, capabilities (e.g. residual energy), status.2. Cluster formation: cluster head will be elected based on the findingsin (1). The cluster children communicate directly with their clusterhead (CH). 3. Route discovery: this task interconnects the electedcluster heads together and finds the route towards the sink smart device(node) so that minimum energy is consumed. 4. Data transmission: themicrocontroller processes the collected color data and based on theadopted data dissemination approach, the smart device 100 will do one ofthe following. (a) Transmit the data as is without considering theprevious status; or (b) transmit the data considering the previousstatus. Here we can have several scenarios, which include: (i)transmitting the data if the change in reported tension exceeds thewarning or emergency levels; and (ii) otherwise, do not transmit.

The electronic of FIG. 2A operates with a big data discovery system ofFIG. 2B that determines events that may lead to failure. FIG. 2B is ablock diagram of an example stress monitoring system 200 that may beprocess the stress detected by the smart device 100 of FIG. 1 , arrangedin accordance with at least some embodiments described herein. Alongwith the stress monitoring system 220, a first smart device such as asmart device 240, a second smart device 250, a third smart device 260, afourth smart device 280, and additional sensors 270 may also beassociated with the unit 200. The stress monitoring system 220 mayinclude, but is not limited to, a transceiver module 222, a stressdetection module 224, a stress prediction module 226, a determinationmodule 228, a stress response module 232, an interface module 234, aprocessor 236, and a memory 238.

The transceiver module 222 may be configured to receive a stress reportfrom each of the first, second, and third sport smart devices 240, 250,260. In some embodiments, the transceiver module 222 may be configuredto receive the stress reports over a wireless network. For example, thetransceiver module 222 and the first, second, and third smart devices240, 250, 260 may be connected over a wireless network using the IEEE802.11 or IEEE 802.15 standards, for example, among potentially otherstandards. Alternately or additionally, the transceiver module 222 andthe first, second, and third smart devices 240, 250, 260 may communicateby sending communications over conductors used to carry electricity tothe first, second, and third smart devices 240, 250, 260 and to otherelectrical devices in the unit 200. The transceiver module 222 may sendthe stress reports from the first, second, and third smart devices 240,250, 260 to the prediction module 226, the stress detection module 224,and/or the determination module 228.

The stress module 224 may be configured to detect stress on the sportobject as detected by the devices 100. The signal sent by the devices100 collectively may indicate the amount of stress being generatedand/or a prediction of the amount of stress that will be generated. Thestress detection module 224 may further be configured to detect a changein stress of non-smart devices associated with the unit 200.

The prediction module 226 may be configured to predict future stressbased on past stress history as detected, environmental conditions,forecasted stress loads, among other factors. In some embodiments, theprediction module 226 may predict future stress by building models ofusage and weight being transported. For example, the prediction module226 may build models using machine learning based on support vectormachines, artificial neural networks, or using other types of machinelearning. For example, stress may correlate with the load carried by abridge or an airplane structure. In other example, stress may correlatewith temperature cycling when a structure is exposed to constant changes(such as that of an airplane).

The prediction module 226 may gather data for building the model topredict stress from multiple sources. Some of these sources may include,the first, second, and third smart devices 240, 250, 260; the stressdetection module 224; networks, such as the World Wide Web; theinterface module 234; among other sources. For example, the first,second, and third smart devices 240, 250, 260 may send informationregarding human interactions with the first, second, and third smartdevices 240, 250, 260. The human interactions with the first, second,and third smart devices 240, 250, 260 may indicate a pattern of usagefor the first, second, and third smart devices 240, 250, 260 and/orother human behavior with respect to stress in the unit 200.

In some embodiments, the first, second, and third smart devices 240,250, 260 may perform predictions for their own stress based on historyand send their predicted stress in reports to the transceiver module222. The prediction module 226 may use the stress reports along with thedata of human interactions to predict stress for the system 200.Alternately or additionally, the prediction module 226 may makepredictions of stress for the first, second, and third smart devices240, 250, 260 based on data of human interactions and passed to thetransceiver module 222 from the first, second, and third smart devices240, 250, 260. A discussion of predicting stress for the first, second,and third smart devices 240, 250, 260 is provided below with respect toFIGS. 5 and 6 .

The prediction module 224 may predict the stress for different amountsof time. For example, the prediction module 224 may predict stress ofthe system 200 for 1 hour, 2 hours, 12 hours, 1 day, or some otherperiod. The prediction module 224 may also update a prediction at a setinterval or when new data is available that changes the prediction. Theprediction module 224 may send the predicted stress of the system 200 tothe determination module 228. In some embodiments, the predicted stressof the system 200 may contain the entire stress of the system 200 andmay incorporate or be based on stress reports from the first, second,and third smart devices 240, 250, 260. In other embodiments, thepredicted stress of the system 200 may not incorporate or be based onthe stress reports from the first, second, and third smart devices 240,250, 260.

The determination module 228 may be configured to generate a unit stressreport for the system 200. The determination module 228 may use thecurrent stress of the system 200, the predicted stress of the system 200received from the prediction module 224; stress reports from the first,second, and/or third smart devices 240, 250, 260, whether incorporatedin the predicted stress of the system 200 or separate from the predictedstress of the system 200; and an amount of stress generated or thepredicted amount of stress, to generate a unit stress report.

In some embodiments, one or more of the stress reports from the first,second, and/or third smart device 240, 250, 260 may contain anindication of the current operational profile and not stress. In theseand other embodiments, the determination module 228 may be configured todetermine the stress of a smart device for which the stress reportindicates the current operational profile but not the stress. Thedetermination module 228 may include the determined amount of stress forthe smart device in the unit stress report. For example, both the firstand second smart device 240, 250 may send stress report. The stressreport from the first smart device 240 may indicate stress of the firstsmart device 240. The stress report from the second smart device 250 mayindicate the current operational profile but not the stress of thesecond smart device 250. Based on the current operational profile of thesecond smart device 250, the determination module 228 may calculate thestress of the second smart device 250. The determination module 228 maythen generate a unit stress report that contains the stress of both thefirst and second smart devices 240, 250.

In some embodiments, the stress monitoring system 220 may not includethe prediction module 226. In these and other embodiments, thedetermination module 228 may use stress reports from the first, second,and/or third smart devices 240, 250, 260, with the received amount ofstress inferred on non-smart devices, if any, to generate the unitstress report. The determination module 228 may send the unit stressreport to the transceiver module 222.

In some embodiments, the processor 236 may be configured to executecomputer instructions that cause the stress monitoring system 220 toperform the functions and operations described herein. The computerinstructions may be loaded into the memory 238 for execution by theprocessor 236 and/or data generated, received, or operated on duringperformance of the functions and operations described herein may be atleast temporarily stored in the memory 238.

Although the stress monitoring system 220 illustrates various discretecomponents, such as the prediction module 226 and the determinationmodule 228, various components may be divided into additionalcomponents, combined into fewer components, or eliminated, depending onthe desired implementation. In some embodiments, the unit 200 may beassociated with more or less smart devices than the three smart devices240, 250, 260 illustrated in FIG. 2 .

FIG. 3 is a flow chart of an example method 300 of monitoring stress ofa sport or game unit, arranged in accordance with at least someembodiments described herein. The method 300 may be implemented, in someembodiments, by an stress monitoring system, such as the stressmonitoring system 220 of FIG. 2 . For instance, the processor 236 ofFIG. 2B may be configured to execute computer instructions to performoperations for monitoring stress as represented by one or more of blocks302, 304, 306, 310, 312, and/or 314 of the method 300. Althoughillustrated as discrete blocks, various blocks may be divided intoadditional blocks, combined into fewer blocks, or eliminated, dependingon the desired implementation.

The method 300 may begin at one or more of blocks 302, 304, and/or 306.The blocks 302, 304, and/or 306 may occur at the same time or atdifferent times and may or may not depend on one another. Furthermore,one or more of the block 302, 304, 306 may occur during the method 300.For example, the method 300 may complete when blocks 304, 310, and 312occurs and without the occurrence of block 302 and 306.

In block 302, a change in stress of a device (device or beam) associatedwith a unit may be detected. A non-smart device may by any device thatreceives stress and does not generate an stress report indicating itsstress, for example a legacy racket without IoT electronics. A change inthe stress of a non-smart device may be detected using an stressdetection module and/or usage meter associated with the unit, such asthe stress detection module 224 and/or the smart device 100. Forexample, non-smart device stress can be estimated by the load the unitcarries, the temperature cycling experienced by the unit, for example.

After a change in stress of the non-smart device is detected, the method300 proceeds to block 310. In block 304, an stress report from a smartdevice such as the smart device 100 associated with the unit may bereceived. A smart device may be a device that detects stress andgenerates and transmits an stress report indicating the stress on thesmart device. The stress report may indicate predicted future stress ofthe smart device. In some embodiments, an stress report may be receivedat set intervals from the smart device regardless of a change in thestress report. Alternately or additionally, a stress report may bereceived after a change in the stress of the smart device results in achange to the stress report. After a stress report is received from thesmart device, the method 300 proceeds to block 310.

In block 306, stress experienced at the unit may be detected. Stress atthe unit may be detected using a stress detection module, such as thestress detection module 224 of FIG. 2B. After detecting stress at theunit, the method proceeds to block 310. At block 310, it is determinedif a change in the stress occurred. For example, if an increase instress occurs at the same time and at the same amount as an increase inthe stress of a non-smart device, a change in the stress may not occur.If a change in the stress occurs, the method 300 proceeds to block 312.If no change occurs, the method 300 ends.

At block 312, a unit stress report is generated for the unit. In someembodiments, the unit stress report may indicate the current stress ofthe unit. Alternately or additionally, the unit stress report mayindicate a current and predicted future stress of the unit. At block314, the unit stress report is transmitted to a maintenance provider. Insome embodiments, the unit stress report may be transmitted when theunit stress report indicates a change in stress for the unit that isgreater than a predetermined threshold. If the unit stress reportindicates a change in stress for the unit that is less than thepredetermined threshold, the unit stress report may not be transmittedto the provider of maintenance services.

FIG. 5 shows in more details the computer 30 and the interface to theprobe 20. An amplifier 90 amplifies vibratory output from a transducer92. A pick up unit having an accelerometer (or an array) 96 receivesreflected vibrations from user arm or leg 94, among others. A computer98 includes a digital converter to digitize output from the pick-up unitand software on the computer 98 can process the captured diagnosticdata. Diagnostic software 100 can include a database of knownrestorations, diseases, and tissue conditions whose signatures can bematched against the capture diagnostic data, and the result can bedisplayed on a screen for review by the athlete.

Included in one embodiment of the instrumentation is the transmitter ortransducer, which will emit the vibrations that will be imparted to theteeth and jaws. This will be connected to a power supply and amplifier,which will allow for a frequency range. On electrical excitation, thetransducer emits an outgoing vibration. That vibration will then travelinto the arm or leg and down is root into the soft tissues and out intothe bones or jaws. The accelerometer or detector will be placed on thebone of interest. It will receive the vibrations from the emitter. Theeffect of the vibrations on the muscle of interest will generate apattern of frequency vibrations. Those vibrations will be digitallyconverted and analyzed against known dental states in the software ofthe computer. As the data is collected various linear samplings andcomparisons will be made against the database. Software will make thesecomparisons as the data is received from the teeth.

FIG. 5 schematically shows a method or app 52 to perform collaborativeVR/AR gaming. The app 52 includes code for:

(51) capture 360 degree view of the live event

(52) detect head position of the viewer

(53) adjust viewing angle on screen based on head position and userposture

(54) render view to simulate action based on user control rather thanwhat the professional is doing

(55) augment view with a simulated object that is powered by vieweraction as detected by sensors on viewer body

(56) compare professional result with simulated result and show resultto a crowd of enthusiasts for social discussion.

FIG. 6 is a flowchart of a method of an embodiment of the presentdisclosure. Referring to FIG. 6 , a smart system may collect from smartdevices state change events of a smart system in operation 601. That is,the smart system of FIG. 4 collects information on each of the group ofdevices, the smart devices, the smart appliances, the security devices,the lighting devices, the energy devices, and the like. The state changeevents indicate when there is a change in the state of the device or thesurrounding environment. The state change events are stored by the smartsystem. In operation 603, the system may determine whether a series ofthe collected state change events are a known pattern. That is, thegateway determines whether there are events which have been correlatedor identified in the past. If the collected state change events havebeen identified in the past, it may be necessary to determine that thesmart systems trusts the identification the collected state changeevents. The trust factor of the identification of the collected statechange events may be determined by the number of users who haveidentified the collected state change events or the number of timecollected state change events have been repeated and identified. Inoperation 605, when the series of the collected state change events isan unknown pattern, request users of the smart system to identify whatcaused the collected state change events request. That is, the systemtransmits to a gamification application (hereinafter app) on the user'smobile device a request to identify the collected state change events.The gamification app displays the information and request the user enterinformation identifying the collected state change events. Each of themobile devices transmits this information back to the system to thegamification module. In operation 605, the system transmits the eachuser's identified collected state change events to the other user's ofthe smart home system and they each vote on the best identification ofthe collected state change events. Thus, the identified collected changestate events that have been repeatedly identified over a period of weeksincreases, the trustworthiness of the identification increases.Likewise, if every user of the smart system makes the sameidentification of the collected change state events, the identifiedcollected change state events may be considered trustworthy at point.Such a determination of a threshold for when the identified collectedchange state events are considered trustworthy and therefore need not berepeated, is made by a system administrator. However, it will beunderstood that such a trustworthiness of this type only gives higherconfidence of this particular dataset at that point in time. As suchfurther repetition is required, since the sensor data may have noise,the more datasets to be identified to the pattern, the more robust thetrustworthiness will be. Until the robustness reaches a threshold, thenthe system can confirm this is a known trustworthy pattern.

The system can use gaming to help sport enthusiasts improve dental careor maintain teeth hygiene. This may involve use of virtual tools,corresponding to such tools used in normal dental hygiene: sport device,tooth picks, dental floss, gum massaging aids, etc. In this embodiment,the game may, for example, have the object of fighting tooth or gumdecay, damage or infection which may be caused by carries or otherinfectious agents. The user is presented with a library of tools and hasto select a tool to treat a certain developing virtual condition, e.g.carries or a gum infection. The game rules determine a certaincontinuous progress of infection which if not properly “treated” by theuser will cause decay of one or more teeth, gum infection, potentialbleeding, loss of teeth, etc. In step 13, the user may score pointsdepending on his ability to choose the right tools to treat a particularcondition or in avoiding a condition from developing. Next, it isdetermined whether the condition of the teeth is satisfactory. If yes,the process terminates. If no, then the user is prompted whether hewishes to select another tool. If no, the process terminates. If yes,the process restarts. Here again, the game, in addition to being amusingand providing an insight of the user into his own teeth, may beeducational, particularly for children, on teeth oral hygiene methodsand on the importance of maintaining oral hygiene.

In accordance with another embodiment of the invention the game mayinvolve use of a variety of virtual imaginary tools such as virtualguns, wands, etc. in order to fight infectious agents of the teeth orgums.

Smart Sport Glove

FIG. 7 shows an exemplary glove which can be thin to provide touchsensitivity or thick to provide shock protection for boxers. A body 12of the boxing glove 10 includes an impact measuring device 14 isembedded within the glove 12 in an area protected from direct impact.Such an area includes the cuff 15 of the glove 12 or that portion of theglove 12 adjacent a user's palm, or adjacent an inside surface of auser's fingers. Placement of the impact measuring device 14 into thelining of the glove in such an area allows for the force of a blow to bemeasured without presenting a hazard to the recipient of the blow. Underthe embodiment, an impact measuring device 14 would be included in theright glove 12 for a right handed fighter, or the left glove 12 for aleft handed fighter. For fighters that are equally effective with bothhands, or to improve monitoring accuracy, an impact measuring device 14would be included in both gloves 12. The impact measuring system 20. Theimpact measuring system 20 includes an impact measuring device 14 andimpact display unit 16. The impact measuring device 14 is linked to theimpact display 28 via a radio frequency (rf) link 32. Under theembodiment, the impact measuring device 14 includes at least one 3-axisaccelerometer. A thin version of the glove can be worn to detect a golfstroke or a tennis stroke with legacy clubs or rackets that lacks IoTintelligence.

Smart Sport Band

FIG. 8 shows an exemplary stick on wearable monitoring device for sportsand fitness applications. The wireless sensor electronics 14 is mountedon a band-aid in the example of FIG. 8 . The band-aid can be removedupon completion of the sports event. The central patch can be recycled,and the adhesive portion can be disposed. While the embodiment is shownas a band-aid, the inventors contemplate that any suitable bands,straps, attachments can be used in lieu of the band-aid to attach thesensors to the body. For example, in Virtual Reality (VR) sportsapplications, sensors including gyroscopes and cameras can be positionedon various body portions to capture motion as well as eye tracking,mouth tracking, speech recognition, among others.

One embodiment uses Samsung's Bio-Processor which is an all-in-onehealth solution chip. By integrating not only Analog Front Ends (AFE),but also microcontroller unit (MCU), power management integrated circuit(PMIC), digital signal processor (DSP), and eFlash memory, it is able toprocess the bio-signals it measures without the need of externalprocessing parts. Even with its integrated design, the Bio-Processor isparticularly innovative thanks to its incredibly small size. Whencompared to the total area of the discrete parts, the Bio-Processor isonly about one fourth of the total combined size, which is ideal forsmall wearable devices, offering a bounty of options when designing newdevices. The Bio-Processor has five AFEs including bioelectricalimpedance analysis (BIA), photoplethysmogram (PPG), electrocardiogram(ECG), skin temperature, and galvanic skin response (GSR) into a singlechip solution that measures body fat, and skeletal muscle mass, heartrate, heart rhythm, skin temperature and stress level, respectively.

One embodiment provides a flexible and stretchable electronic patch thatmonitors impact or other events whereby a flexible substrate isgeometrically patterned to allow the substrate to undergo substantialstretching and flexing while large regions of the substrate materialexperiences local strains much lower than the macroscopic appliedstrain. The geometric patterning of the substrate facilitates continuouslow strain domains (LSDs) throughout the substrate—where low straindomains are defined as regions that experience strain levels (magnitude)lower than the macroscopic applied strain. Conventional electroniccomponents can be mounted to the LSDs, and conventional metal traces canbe routed through the LSDs, dramatically reducing the stressestransmitted to the components and traces by the substrate duringstretching and flexing, and therefore reducing the potential forcomponent debonding, trace cracking, and circuit failure. Thegeometrically patterned strain relief features (SRFs) are dispersedeither regularly or irregularly throughout the substrate. Thegeometrically patterned SRF regions form “hinge-like” domains. Duringmacroscopic deformation, the SRFs rotate, translate, open, close, orotherwise change shape, causing the “hinge-like” regions to deform, andthe remaining larger LSD substrate regions to primarily rotate andtranslate. The SRFs are designed such that the “hinge-like” regions alsoexhibit relatively small strain compared to the macroscopic appliedstrain and thus enable conductive traces, such as copper or gold, to runthrough the hinges and maintain function during stretching, flexing andtwisting of the patch. The substrate can be multilayered to enablerunning conductive traces, ground layers, vias, and/or components on/inmultiple layers through the thickness of the overall substrate. Thegeometric patterning can be designed to enable different stretching,flexing and twisting, providing uniaxial, biaxial, and multi-axialstretchability or flexibility, and the ability to conform to a varietyof surface curvatures. The geometrically patterned substrate offers ameans of packaging complex multi-layered electronics designs formonitoring impact (and other) events onto a stretchable and flexiblesubstrate enabling the device to dynamically stretch, bend, twist, andconform to arbitrary shapes. The stretchable, flexible geometricallystructure electronics can be fabricated using the same technologies forconventional flexible circuit boards where the stretch-enablingpatterning can be imparted at different stages in the fabricationprocess and can also be fabricated using emerging materials andfabrication methods. The Stretchable bandaid has the stretchable,flexible substrate described above with multiple LSDs for placement ofelectronic components (e.g., accelerometers, gyroscopes, pressuretemperature, gas and fluid sensors, microprocessors, transceivers, GPS,clocks, actuators, vias, and batteries (or other energy source)) andmultiple patterned hinge-like regions bridging the LSDs which enable therouting of conducting interconnecting traces. The SEHIM patch can takethe form factor of a bandaid or bandage or other such wearable formfactor. The geometric patterning provides stretch, flex and twist toconform to a body and stretch, flex and twist to move or deform with abody. The bandaid detects impact accelerations, using a 3-axisaccelerometer and processes the raw acceleration data in themicroprocessor. The processed data is stored in the microprocessor andlater (or potentially in real time) transmitted via the Bluetooth to asmart phone, tablet or computer. This embodiment encompasses wirelesscommunication but wired communication may be desirable in someapplications and can be accommodated by this invention. The bandaid canbe stretched, bent and twisted with the traces and components at lowstrains to maintain electrical function. In all cases there waseffectively no strain on the components and solder joints. The bandaidcan also possess an adhesive backing for direct adhesion to the head,body or object. The band can also be coated to provide both addedcomfort and protection against moisture, water, and other environmentalfactors. The band can also contain other sensors including gyroscopes,temperature and pressure sensors, moisture sensors, clocks, chemicaland/or biological sensors, etc.

Smart Clothing

FIG. 9 shows an exemplary shirt based embodiment where sensors can bepositioned anywhere on the shirt and when worn, can capture position,video, and vital signs. One embodiment uses Samsung's Bio-Processor toprocess the bio-signals it measures without the need of externalprocessing parts with five AFEs including bioelectrical impedanceanalysis (BIA), photoplethysmogram (PPG), electrocardiogram (ECG), skintemperature, and galvanic skin response (GSR) into a single chipsolution that measures body fat, and skeletal muscle mass, heart rate,heart rhythm, skin temperature and stress level, respectively. Featuresof the smart clothe can include:

1. A smart clothing, comprising:

-   -   a shirt, underwear, pant or sock;    -   a band to be secured to the a shirt, underwear, pant or sock;    -   a processor in the band and coupled to a wireless transceiver;

an EKG amplifier coupled to the band;

a sensor disposed in the band; and

an accelerometer disposed within the band to detect acceleration of theband.

2. The clothing, comprising a plurality of bands forming a mesh networkand communicating episodically to conserve power.

3. The clothing where the electronic components, sensors, andinterconnects of the patch monitor, record, process and/or transmitevents of interest (such as accelerometers and gyroscopes for impactevents, temperature sensors for temperature and/or temperaturegradients, pressure sensors, moisture sensors, chemical sensors).

4. The clothing comprised for sensing and/or monitoring impact eventswhere the sensors are accelerometers, gyroscopes, and/or pressuresensors.

5. The clothing comprised for sensing and/or monitoring and/orcontrolling ongoing events where the sensors monitor temperature,temperature gradients, motion, position, environmental or chemicallevels, or other such information.

6. The clothing comprised for sensing events or other informationincluding mounting multiple distributed sensors for obtaining spatialand/or temporal distribution in the data and/or multiple sensors sensingdifferent information and data.

7. The clothing including wired or wireless communication, such as aBluetooth module or a wi-fi module or other transmission module,transmitting and/or receiving information to/from another device.

8. The clothing with power and energy sources including batteries, wiredor wireless rechargeable batteries, photovoltaics, thermoelectrics, orenergy harvesters.

9. The clothing with an adhesive backing for directly adhering to ahead, a body, or an object.

10. The clothing contained in an adhesive or a sleeve for adhering orattaching to a head, a body, or an object.

11. The clothing coated with a coating for protection against theelements (water, moisture, dirt, other) and/or for increased comfort tothe wearer.

12. The clothing, comprising a geometrically patterned substrate thatcontains regions of low strain domains (LSDs) bridged by hingeablestrain relief features (SRFs) which also contain low strain regions andenable the stretching, flexing and twisting of the patch whilemaintaining continuous low strain regions for mounting electroniccomponents and routing traces.

13. The clothing for attachment to or on or an object, or embedded in anobject.

14. The clothing in the form factor of a rectangular or a square or atriangular or other polygon or circular or elliptical or other geometricshape bandage.

15. The clothing in the form factor that is or contains any combinationof rectangles, triangles, circles, ellipses or other form factors.

16. The clothing with different geometric patterning of differentnumbers and shapes and orientations of low strain domains, differentnumbers and orientation of geometrically structured hinge-like domains,and different geometries of hinge-like domains.

17. The clothing as a programmable circuit board for arbitraryapplications.

18. The clothing fabricated using current flex circuit manufacturingmethods and materials.

19. The clothing comprising a cloud storage to receive sensor data.

20. The clothing where the polymer layers are current flex manufacturingpolymers such as Kapton, polyimides, polyamides, polyesters, or other aswell as elastomers such as silicone rubbers (PDMS) or polyurethanes orother elastomers and the interconnects are metals that have highelectrical conductivity, such as copper or gold, or where theinterconnects are emerging stretchable electronic materials andstretchable conductive inks and materials.

Smart Handle

FIGS. 11A-11B show an exemplary smart handle for sports such as tennis,badminton, table tennis, and golf, among others. The wireless sensorelectronics 14 is mounted on a handle in the example of FIG. 11B. Thehandle can be embedded or can be removed upon completion of the sportsevent. The sports event does not have to be real, for example, inVirtual Reality (VR) sports applications, sensors including gyroscopesand cameras can be positioned on various body portions to capture motionas well as eye tracking, mouth tracking, speech recognition, amongothers.

The handle includes a swing analyzer measurement portion 54 in the gripend 52 of the handle of a golf club or a tennis/badminton racket, and aremote or handheld unit 56. The swing analyzer measurement portion 54includes an accelerometer 16 of combination accelerometer and gyroscopeor magnetometer unit, a processor unit 58 coupled to the accelerometer16, and a battery 20 that is electrically coupled to and provides powerto the accelerometer 16 and processor unit 58. A camera is included tocapture videos of the swing and also the game in progress for futurereference. A communications unit 60 is also housed in the grip end 52 ofthe golf club 50, receives power from the battery 20, and is coupled tothe processor unit 58. Swing analyzer measurement portion 54, with orwithout the communications unit 60, may be assembled as an integral unitand inserted into a hollow portion of the handle of the golf club ortennis/racket handle 50 at the grip end 52 thereof. Processor unit 58may be an integrated device that includes hardware and softwarecomponents capable of processing acceleration measured by theaccelerometer(s) 16 and converting the measured acceleration into dataabout the force on the shaft and position of the face of the club atimpact at a set distance. If the measured force exceeds a threshold themeasured force or a signal derived therefrom is transmitted via thecommunications unit 60 to the handheld unit 56. If not, acceleration andface position at impact of the golf club or tennis racket handle 50 isobtained again. The threshold is set so that only acceleration or forcemeasurements arising from actual swings of the golf club 50 aretransmitted to the handheld unit 56. Handheld or remote unit 56 includesan application or computer program embodied on a non-transitorycomputer-readable medium that performs the golf ball carrying distanceestimation or prediction steps, as well as manages the training stagedescribed above. Importantly, the handheld unit 56 receives accelerationmeasurement data from the golf clubs/tennis rackets equipped with aswing analyzer measurement portion 54 and the club face angle inrelation to the swing plane, and manages the carrying distanceestimation steps for all golf clubs equipped with the swing analyzermeasurement portion 54 that are designed to communicate therewith.Handheld or remote unit 56 may be a standalone unit for use only withthe golf clubs equipped with the swing analyzer measurement portion 54,and incorporating the application thereon, or may be a smartphone orsimilar device with the application embodied thereon or downloadedthereto and that can be used for other purposes. Handheld or remote unit56 includes a communications unit 70 that communicates with thecommunications unit 60 on each golf club or tennis racket handle 50,i.e., with the communications units present on all of the golf clubs 50equipped with swing analyzer measurement portions 54 and which have beendesignated to communicate therewith. Communications unit 70 may be anintegral part of the handheld unit 56 as is the case when the handheldunit 56 is a smartphone. Communications unit 70 may also communicatewith another device such as a Smartphone, to perform more datamanipulations relating to the golf swing and/or swing results to providemore information to the user. The data and the calculation/manipulationresults can be stored in the Smartphone and displayed when desired.Currently usable Smartphones are Apple iOS iPhones and Android operatingsystem phones. Handheld or remote unit 56 also includes a processor unit72, a storage unit 74 and a display 76. When the handheld unit 56 is asmartphone or similar device, all of the processor unit 72, storage unit74 and display 76 may be integral components thereof. Processor unit 72performs functions similar to those performed by the processor unit 18described above, e.g., calculates an estimated carrying distance for thegolf ball based on the acceleration measured by the accelerometer(s) 16and transmitted via the communications units 60, 70, and the type ofclub provided to the application or computer program in the processorunit 72. Storage unit 74 receives and stores information about thecarrying distance of each club as a function of clock or swing position,e.g., in the form of a virtual table associating the type of club, theswing or swing position and the estimated carrying distance.

Other sensors can be used as well. For example, the handle can containconductive ink to capture biometric. One embodiment uses Samsung'sBio-Processor which is an all-in-one health solution chip to measurebioelectrical impedance analysis (BIA), photoplethysmogram (PPG),electrocardiogram (ECG), skin temperature, and galvanic skin response(GSR) into a single chip solution that measures body fat, and skeletalmuscle mass, heart rate, heart rhythm, skin temperature and stresslevel, respectively. The handle can also contain other sensors includinggyroscopes, temperature and pressure sensors, moisture sensors, clocks,chemical and/or biological sensors, etc. Features of the smart handlecan include:

Smart Protective Gear

FIGS. 12A-12C illustrate smart protective gears embedded with the IoTsensors and instrumentations to report potential health issues. Forsoccer, the protection includes shin guards. For football, theprotection includes Helmets, Chin Straps & Chin Shields, Cups & AthleticSupporters, Elbow Sleeves & Arm Pads, Back Plates & Rib Protection,Facemasks, Girdles, Helmet Visors, Shoulder Pads, Hip & Tail Pads,Mouthguards, Neck Rolls. For motorcycling, the protection includeshelmet, should pads, jacket with back protection, padded gloves, leatherpants, knee pads, and boots. For rock climbing, the protection includesshoes, carabiners, webbing, harnesses, among others.

The wireless sensor electronics 14 is mounted on the helmet or shoulderpad in the example of FIG. 12A or 12C. The electronics 14 can beembedded or can be removed upon completion of the sports event. Thesports event does not have to be real, for example, in Virtual Reality(VR) sports applications, sensors including gyroscopes and cameras canbe positioned on various body portions to capture motion as well as eyetracking, mouth tracking, speech recognition, among others.

The protection gear includes an impact sensor such as an accelerometerto indicate if concussion has occurred. Other sensors can be used aswell. For example, the handle can contain conductive ink to capturebiometric. One embodiment uses Samsung's Bio-Processor which is anall-in-one health solution chip to measure bioelectrical impedanceanalysis (BIA), photoplethysmogram (PPG), electrocardiogram (ECG), skintemperature, and galvanic skin response (GSR) into a single chipsolution that measures body fat, and skeletal muscle mass, heart rate,heart rhythm, skin temperature and stress level, respectively. Thehandle can also contain other sensors including gyroscopes, temperatureand pressure sensors, moisture sensors, clocks, chemical and/orbiological sensors, etc.

Impact sensors, or accelerometers, measure in real time the force andeven the number of impacts that players sustain. Data collected is sentwirelessly via Bluetooth to a dedicated monitor on the sidelines, whilethe impact prompts a visual light or audio alert to signal players,coaches, officials, and the training or medical staff of the team. Onesuch sensor example is the ADXL377 from Analog Devices, a small, thinand low-power 3-axis accelerometer that measures acceleration frommotion, shock, or vibration. It features a full-scale range of ±200 g,which would encompass the full range of impact acceleration in sports,which typically does not exceed 150 g's. Specifically designed forconcussion and head-trauma detection, at 3 mm×3 mm×1.45 mm, the deviceis small enough to be designed into a helmet. Sensitivity, listed at 6.5mV/g with −3 dB bandwidth at 1.6 kHz, is sufficiently high for theapplication. When a post-impact player is removed from a game and notallowed to return until cleared by a concussion-savvy healthcareprofessional, most will recover quickly. If the injury is undetected,however, and an athlete continues playing, concussion recovery oftentakes much longer. In addition, the industry is finding that long-termproblems from delayed or unidentified injury can include: Earlydementia, Depression, Rapid brain aging, and Death. The cumulativeeffects of repetitive head impacts (RHI) increases the risk of long-termneuro-degenerative diseases, such as Parkinson's disease, Alzheimer's,Mild Cognitive Impairment, and ALS or Lou Gehrig's disease. The sensors'most important role is to alert to dangerous concussions. Yet, the actof real-time monitoring brings these players to the attention of theircoaches not only to monitor serious impacts but, based on the dataprovided by the sensors, also help to modify a player's technique sothat they are not, for example, keeping their head low where they cansustain injury to the front and top of the skull. In the NFL there alsohas been an aggressive crackdown against hits to the head and neck—aresponse to the ongoing concussion crisis—resulting in immediate penaltyto players using their helmets as a “weapon”. Customized mouthguardsalso have sensors therein. A customized mouthguard has tested to be 99percent accurate in predicting serious brain injury afternear-concussive force, according to an Academy of General Dentistrystudy2. Teeth absorb and scatter infrared light, which shows how muchforce is taking place at the moment of impact.

Features of the smart protective gear can include:

1. A smart protection gear, comprising:

-   -   a wearable surface;    -   a processor in the band and coupled to a wireless transceiver;    -   a camera coupled to the surface;    -   a sensor disposed in the surface; and

an accelerometer disposed within the band to detect acceleration of thesurface.

2. The protection gear, comprising a plurality of smart protection gearsforming a mesh network and communicating episodically to conserve power.

3. The protection gear where the electronic components, sensors, andinterconnects of the protection gear monitor, record, process and/ortransmit events of interest (such as accelerometers and gyroscopes forimpact events, temperature sensors for temperature and/or temperaturegradients, pressure sensors, moisture sensors, chemical sensors).

4. The protection gear comprised for sensing and/or monitoring impactevents where the sensors are accelerometers, gyroscopes, and/or pressuresensors.

5. The protection gear comprised for sensing and/or monitoring and/orcontrolling ongoing events where the sensors monitor temperature,temperature gradients, motion, position, environmental or chemicallevels, or other such information.

6. The protection gear comprised for sensing events or other informationincluding mounting multiple distributed sensors for obtaining spatialand/or temporal distribution in the data and/or multiple sensors sensingdifferent information and data.

7. The protection gear including wired or wireless communication, suchas a Bluetooth module or a wi-fi module or other transmission module,transmitting and/or receiving information to/from another device.

8. The protection gear with power and energy sources includingbatteries, wired or wireless rechargeable batteries, photovoltaics,thermoelectrics, or energy harvesters.

9. The protection gear with an adhesive backing for directly adhering toa head, a body, or an object.

10. The protection gear contained in an adhesive or a sleeve foradhering or attaching to a head, a body, or an object.

11. The protection gear coated with a coating for protection against theelements (water, moisture, dirt, other) and/or for increased comfort tothe wearer.

12. The protection gear, comprising a geometrically patterned substratethat contains regions of low strain domains (LSDs) bridged by hingeablestrain relief features (SRFs) which also contain low strain regions andenable the stretching, flexing and twisting of the protection gear whilemaintaining continuous low strain regions for mounting electroniccomponents and routing traces.

13. The protection gear for attachment to or on or an object, orembedded in an object.

14. The protection gear in the form factor of a rectangular or a squareor a triangular or other polygon or circular or elliptical or othergeometric shape bandage.

15. The protection gear in the form factor that is or contains anycombination of rectangles, triangles, circles, ellipses or other formfactors.

16. The protection gear with different geometric patterning of differentnumbers and shapes and orientations of low strain domains, differentnumbers and orientation of geometrically structured hinge-like domains,and different geometries of hinge-like domains.

17. The protection gear as a programmable circuit board for arbitraryapplications.

18. The protection gear fabricated using current flex circuitmanufacturing methods and materials.

19. The protection gear comprising a cloud storage to receive sensordata.

20. The protection gear where the polymer layers are current flexmanufacturing polymers such as Kapton, polyimides, polyamides,polyesters, or other as well as elastomers such as silicone rubbers(PDMS) or polyurethanes or other elastomers and the interconnects aremetals that have high electrical conductivity, such as copper or gold,or where the interconnects are conductive inks.

Custom Gear

In one aspect, the protective gear is custom formed to the athlete'sbody. This is done in FIG. 12C as follows:

-   -   321) perform 3D scan of person and create 3D model    -   322) form positive mold from the 3D model    -   323) place mold into 2 phase 3D printer to form a negative    -   324) put composite material into mold and form composite        protection gear    -   325) embed IoT electronics into one or more locations into the        composite protection gear    -   326) link IoT electronics with mobile devices and cloud based        storage and process impact data and warn user if impact is        unsafe.

The protection gear or footwear can be custom produced at the request ofa customer, who can specify the nature of the customization for one ormore pairs of helmet, protective gear, or footwear. Each helmet of thefootwear may have a different design, message or message portiondesigned into it and rendered using the bed of pins described below tomake the custom helmet or shoe design messages or shapes, and then thebottom sole can be fabricated using the reformable bed described below.Once the negative is fixed in the reformable bed, suitable materials forthe bottom sole can be deposited and cured and can include rubber,plastic, or foam. Further customization can be done by a ComputerizedNumerical Control (CNC) where component design can be integrated withcomputer-aided design (CAD) and computer-aided manufacturing (CAM)programs. The device can be programmed to use a number of differenttools-drills, saws, and so on. Alternatively a number of differentmachines can be used with an external controller and human or roboticoperators that move the component from machine to machine. Regardless, aseries of steps needed to produce a part can produce a part that closelymatches the original CAD design in a highly automated fashion. Inaccordance with aspects of the subject matter disclosed herein throughthe use of reformable bed and a suitably programmed CNC tools,customized footwear with custom cut sole designs, can cost effectivelybe created in small quantities and yet scalable for mass-customization.

Shock Protection

In one embodiment, the sole is not completely filled with material, butis formed as a lattice structure. The system generates triangulatedsurfaces for export to additive manufacturing (AM) processes.Implementing a process that coverts a CAD object into an image, known asvoxelisation, the company uses an image-based method which allowsdesigners to generate implicitly defined periodic lattice structuressuitable for additive manufacturing applications and finite elementanalysis (FEA). The system generates robust lattice structures canovercome the problems faced with hollowing out a part to reduce weightand optimize designs prior to 3D printing. Cellular lattice structurescan be used to replace the volume of CAD and image-based parts, reducingweight whilst maintaining optimal performance. In this way, the shoescan be light weight yet strong and provide shock impact absorptionduring running for the wearer.

Topology optimization can be used to drive the material layout includingthe lattice regions. From this new topology optimization implementation,the system can identify void regions in the design space, where thematerial can be removed, regions where solid material is needed, andregions where lattice structure is required. This allows the system togenerate the optimal hybrid or blended solid-lattice design based ondesired functionality of the part.

Lattice structures can be considered as porous structures. In the caseof topology optimization, the semi-dense elements are like the porousmedia. To refine the design, a second-phase involves a detailed sizingoptimization where the end diameters of each lattice cell member areoptimized. This allows for further weight reduction while meeting designrequirements, such as buckling, stress, and displacement.

A piezo material can be actuated to generate a vibration that cancelsincoming shock on the wearer. In one embodiment, the system tracks theshock such as the foot contact patterns and generates an anti-vibrationsignal to cancel the shock generated when the foot contacts the ground.In this embodiment, a processor receives foot ground contact using anaccelerometer. The stride pattern is determined, and the next footground contact is detected, and the piezo material is actuated with acounter signal to cancel the expected shock. This is similar to thenoise cancellation, except the vibration/shock is canceled.

In one hybrid embodiment, the shoes incorporate passive and activeisolation elements. The passive component consists of springs whichsupport the load weight and provide isolation over a broad spectrum.These springs provide a basic level of isolation in the lowerfrequencies and excellent isolation in the higher frequencies (above 200Hz). They also support the load while allowing for travel of theactuators in the active component. The performance of the springs isaugmented and corrected by an active isolation component. The activeisolation component consists of vibration sensors, control electronics,and actuators. The vibration sensors are piezo accelerometers. Aplurality of sensors in each isolation system are positioned indifferent orientations to sense in all six degrees of freedom. The piezoaccelerometers convert kinetic vibration energy into electrical signalswhich are transmitted to the control electronics. The electronicsreconcile and process the signals from the various sensors using aprocessor. The electronics then send a cancellation signal to theactuators. The actuators generate vibrations that are equal to theincoming vibrations but out of phase in relation to the incomingvibrations. This results in cancellation of the incoming vibrationalnoise, leaving the wearer undisturbed. This process occurs within 5-20milliseconds of a vibration entering the system.

Smart Kitchen

FIG. 13A shows an exemplary kitchen system that communicates with theInternet and interoperates with each other to provide the best homeexperience for consumers. The system includes Internet enabled cookingproducts such as ranges, dishwashers, disposers and compactors, waterfilters, hoods and vents, grills, food processors, blenders,refrigerators, slow cookers and multi-cookers, stand mixers, coffeemakers, waffle bakers, toasters, microwave ovens, and countertop ovens,among others. These appliances communicate over a wireless network suchas WiFi, Zigbee, or Bluetooth, for example.

Cameras and sensors can be used in these exemplary IOT appliances toprovide intelligence. In one embodiment, an acoustic sensor listens tothe sound of popping and when that slows down it reduces or stops heatenergy being applied to the item being cook. The appliances can have“Doneness” sensors that are supposed to determine when your food isdone. One sensor is a humidity sensor that is based on the generation ofmoisture vapor from the food, and the humidity sensor then shuts off themicrowave one a certain level of humidity is reached inside the cookingcavity. Another type is a temperature sensor that is actually atemperature probe that plugs into the receptacle in the wall of the ovenand manually pushed into the food to measure the temperature. The userthen sets a finish temperature, let's say 160° F. for meat or poultry,and when the temperature is reached the oven shut off. The camera candetermine the color of the item being cooked, and when a particularcolor is reached, the camera can change the temperature of the oven foroptimum flavor/taste, among others. The camera can be used in the dishwashers to detect difficult spots to be washed and aim additionalcleaning power to the difficult spots. The camera can also inspect thefood or clothing and move the food or clothing to provide better cookingor cleaning power for the specific item's configuration, for example.

An electronic nose can be used that includes a fan module, a gasmolecule sensor module, a control unit and an output unit. The fanmodule is used to pump air actively to the gas molecule sensor module.The gas molecule sensor module detects the air pumped into by the fanmodule. The gas molecule sensor module at least includes a gas moleculesensor which is covered with a compound. The compound is used to combinepreset gas molecules. The control unit controls the fan module to suckair into the electronic nose device. Then the fan module transmits anair current to the gas molecule sensor module to generate a detecteddata. The output unit calculates the detected data to generate acalculation result and outputs an indicating signal to an operator orcompatible host computer according to the calculation result. The nosecan be used in ovens to detect proper cooking targets to adjust and/orshut off temperature in the oven. The nose can be used in refrigerationchambers to detect malodors indicating rotting food that needs removal.Similarly, the nose can be used in disposers and compactors to detectmalodors indicating rotting food that needs removal. In the garage, a COsensor can be used to detect harmful air. The nose can detectbaking/cooking activities in the oven and automatically adjust the speedof the fan in the hood/vent to remove food smell from the house. Thenose can be used in the ovens, slow cookers, grills, countertop ovens,microwave ovens, toasters, waffle bakers, coffee makers, or evenrefrigerators.

An electronic tongue sensor can be provided to sense quality of foodand/or water. The tongue includes a stirring module, a liquid moleculesensor module, a control unit and an output unit. The stirring module isused to pump liquid actively to the liquid molecule sensor module. Themolecule sensor module detects the liquid molecules pumped into by thestirring module. The liquid molecule sensor module at least includes amolecule sensor which is covered with a compound. The compound is usedto combine preset liquid molecules. The control unit controls thestirring module to pump liquid to be “tasted” into the electronic tonguedevice. Then the module transmits a flow current to the liquid moleculesensor module to generate a detected data. The output unit calculatesthe detected data to generate a calculation result and outputs anindicating signal to an operator or compatible host computer accordingto the calculation result. Such electronic tongue can detect quality offood or liquid, among others. The tongue can be used in the ovens, slowcookers, grills, countertop ovens, microwave ovens, toasters, wafflebakers, coffee makers, food processors, blenders, mixers, orrefrigerators.

A fullness sensor can be used to compact the disposers and/or signal theuser to remove the full trash container, for example. The water filterscan have sensors that detect end of life for the filters.

A voice sensor can be used to interface the appliance with the user.User speech is encoded into a compact digital form which is communicatedwith a server, loaded with a series of models honed to comprehendlanguage. The user speech may be evaluated locally, on the appliance. Arecognizer installed on the mobile device communicates with that serverin the cloud to gauge whether the command can be best handledlocally—such as if the user wanted a particular cooking temperature—orif it must connect to the network for further assistance. The servercompares the user speech against a statistical model to estimate, basedon the sounds spoken and the order of the verbal command, what lettersmight constitute it. (At the same time, the local recognizer comparesthe speech to an abridged version of that statistical model.) For both,the highest-probability estimates get the go-ahead. The user speech—nowunderstood as a series of vowels and consonants—is then run through alanguage model, which estimates the words that the speech is comprisedof Given a sufficient level of confidence, the computer then creates acandidate list of interpretations for what the sequence of words in theuser speech might mean. Thus, if the user asks for wines that may gowell with a particular food being cooked, the system searches theinternet for matching wines and reply.

The appliances can be remotely controlled using one or more recipes. Therecipes can be manually entered, or can come from a subscriptiondatabase where users pay to access content. In one embodiment, exoticfood from different cultures can be downloaded and the appliance canautomatically apply predetermined settings to cook food as expertswould. In one embodiment with a smart blender, a prepackaged containerhas a plurality of chambers, each containing different materials to beblended. The recipe can specify that the chambers open in apredetermined manner, and the mixing speed can be adjusted for eachmaterial to optimize the blends to arrive at an expertly made drink.

In another embodiment for oven cooking, a prepackaged container has aplurality of chambers each containing a different ingredient to bereleased in a predetermined sequence and following a predeterminedtemperature profile. The result is a fresh food product that issignificantly better than existing canned or dried food where everythingis done at the factory without access to fresh ingredients.

The recipes, or other information are collected in a database on one ormore servers. The server is connected to the Internet, whereupon itreceives various recipes or other information from a number of computingdevices or websites, which are also connected to the Internet. Thereceived information is converted into records having a consistentrecord format and is stored in the database. A selection of the recordspicked by a user, e.g., according to a subscription, is thencommunicating to the blender appliance. This can be achieved by sendingthe selection of the records to the user's computing device which needsto also be connected to the Internet, or directly to the computingdevise imbedded in the blender appliance which may also be connected tothe Internet. Alternatively, the selection of the records may be storedon medium readable by the computing device imbedded in the blender,e.g., flash memory, diskettes, CDs, etc.

Once the selection of the records is loaded into or imbedded in theappliance of the invention, a choice of various criteria for organizingsuch a selection of records is presented on the display screen of theblender appliance, and the user, using navigational keys of thecomputing device imbedded in the blender appliance, chooses the criteriafor sorting of the records. The records are then sorted according to thechosen criteria and the sorted records are listed on the display screen.Using the navigational keys the user chooses at least one recipe anddisplays it on the display screen. After viewing a chosen recipe theuser may display a next or previous recipe from the sorted list ofrecipes using the navigational keys.

The smart appliances are able to operate with minimal human supervision.For example, if the oven detects over boiling or over cooking, the ovenwould automatically reduce heat to save the food being cooked. Inanother example, if the refrigerator detects that vegetable is rotting,the refrigerator sends a message to the owner to alert them of spoilage.The user can send an instant message to control and communicate with theappliances remotely. The user can check contents of the refrigerator,download family-pleasing recipes to the range, turn on yourwasher/update cycles, and command a robot vacuum to clean the livingroom carpet and kitchen floor. And all this can be accomplished with orwithout user intervention. Should problems arise with the appliance, adiagnosis helps troubleshoot issues quickly and efficiently. Otherkitchen appliances include slow cookers, pasta makers, food processors,bread makers, small ovens, toaster ovens, and the like.

In one embodiment, the appliances interoperate with each other using aninteroperable multivendor appliance protocol that is more than justBluetooth/WiFi. For example, the interoperable multivendor applianceprotocol allows the refrigerator to communicate with smart packages inthe refrigerator and when the package is removed by the user, thecooking instructions are automatically communicated from therefrigerator to a suitable appliance, for example a blender or amicrowave oven or a grill cooker.

Embodiments of the appliances provide social applications for marketingand advertising that can transform the way companies engage withcustomers, analyze their behavior, and optimize the impact of theirinteractions. Examples include location-based services, viral marketing,and mobile advertising. The IoT appliances create and implement cohesivemarketing and advertising strategies across numerous and disparatechannels (TV, radio, Internet, point of sale). IoE will enable companiesto have a complete view of their customers (behaviors, preferences,demographic profile) and deliver individually targeted messages andoffers to them on any device at the time and location where they willhave the most beneficial impact. Thus, companies can react more quicklyby assessing and reacting to their markets in real time; increaseprofits by offering pricing based on customers' situation and ability topay; and grow revenues by bundling their offerings with other productsand services based on a holistic assessment of customers' wants andneeds in the comfort of their homes.

High power appliances such as ovens, washers, and refrigerators cancommunicate expected electrical usage back to electric utilities, andsuch information can improve the electric grid by automaticallydetecting and repairing problems, controlling electrical flows based onreal-time demand, improving generator utilization, and enabling moresustainable energy sources such as wind and solar power. In oneembodiment, the refrigerator precharges its cool temperature at nightwhen energy is less expensive so that the energy consumption during theday is reduced. This is done in one embodiment by freezing salted watercontainers in the refrigerator side panels at night and allowing thefrozen salted containers to change phase and reduce temperature as theice thaws. Similarly, the washers can be set to run at night to reducegrid demand during the day.

A module may be further configured to handle reduced usage requests froman electricity monitoring system associated with the smart device 500.In some embodiments, the reduced usage request may direct the smartdevice to operate using a specific operational profile, such as apower-off, a standby, or a suspend operations profile. Alternately oradditionally, the reduced usage request may contain the requestedelectricity reduction amount and may allow the determination module toselect an operational profile to meet the requested electricityreduction. A transceiver module may be configured to send an electricityusage report to an electricity monitoring system and to receive areduced usage request from an electricity monitoring system. Thetransceiver module may also be configured to send data regarding humaninteractions with the smart device to an electricity monitoring systemfor a unit with which the smart device 500 is associated. In someembodiments, the transceiver module 520 may send and receive data over awireless network according to a given standard, such as, IEEE 802.11,IEEE 802.15, or some other standard. Alternately or additionally, thetransceiver module may send and receive data over conductors used tocarry electric power for a unit with which the smart device isassociated.

A prediction module may be configured to predict future electricityusage of the smart device based on current and past user interactionswith the smart device as collected by the interface module,predetermined electricity usage for each operational profile,operational sub-profile, and/or selected range of operations for thoseoperational profiles and sub-profiles for the smart device, datareceived from an electricity monitoring system associated with the smartdevice, data from networks, such as the World Wide Web, and othersources of information. The data from the electricity monitoring systemmay include human behaviors regarding other devices that may beassociated with the smart device or general human behaviors with respectto electricity usage of a unit with which the smart device isassociated. The prediction module 560 may function similar to theprediction module. The prediction module may send the predictedelectricity usage for the smart device to the determination module forinclusion in the electricity usage report. In some embodiments, theprediction module may predict future electricity usage by buildingmodels of electricity usage of the smart device. For example, theprediction module may build models using machine learning based onsupport vector machines, artificial neural networks, or other types ofmachine based learning using the above-described types of data.

As an example of the operation of the prediction module, the predictionmodule may determine a predetermined electricity usage for a selectedoperational profile and range of operation for the smart device. Basedon past user interactions, the prediction module may predict that theuser may not have the smart device perform the selected operationalprofile for the full range of operation. The prediction module 560 mayreduce the electricity usage from the predetermined amount accordingly.

For example, the smart device may be a microwave. The user may select acooking time of one minute. Historically, when a user selects a cookingtime of one minute, the user may stop the microwave after only 15, 30,or 45 seconds. The prediction module may predict that the user will stopthe microwave before the microwave operates for the full one minute andthus determine electricity usage less than the predetermined amount forone minute of cooking. As another example, the smart device may be acable box. The user may select a show to watch that is one hour long.Historically, when a user selects a show of one hour, the user may watchthe entire show; however, on Fridays between 7 and 8 o'clock, the usermay only watch a show for an average of 35 minutes. On Fridays, theprediction module 560 may predict that the user will watch for 35minutes and thus determine electricity usage less than the predeterminedamount for a one-hour show. In some embodiments, the processor may beconfigured to execute computer instructions that cause the smart deviceto perform the functions and operations described herein. The computerinstructions may be loaded into the memory for execution by theprocessor and/or data generated, received, or operated on duringperformance of the functions and operations described herein may be atleast temporarily stored in the memory. The smart device as discussedabove may provide for various advantages. For instance, a manufacturerof the smart device may not need to disclose device operations to allowan outside device, such as an electricity monitoring system, todetermine electricity usage for a selected operational profile becausethe smart device may determine and send its electricity usage and/orpredicted electricity usage. Alternately or additionally, when the smartdevice is upgraded or the smart device is newly associated with a unit,an electricity management system associated with the unit does not needto be upgraded or changed to determine the electricity usage of thesmart device because the smart device may report its electricity usageto the electricity management system.

Although the smart device illustrates various discrete components, suchas the prediction module and the determination module, variouscomponents may be divided into additional components, combined intofewer components, or eliminated, depending on the desiredimplementation.

A robotic system is provided to dexterously transfer cooking itemsto/from the appliances. The hand can be mounted on a home robot or aconveyance system to pick up cooking ingredients and to move them to acutting board and then to an oven if heat is needed. The robot includesa plurality of cameras, and images are processed by a vision system suchas OpenCV to detect objects and the handle them. The robotic hand isattached to a wrist section which is movably connected to a forearmsection. The robotic hand includes five moveable fingers that closelyresemble the fingers on a human hand and are capable of moving indirections having a total of twelve degrees of freedom. Each of the twodexterous fingers and the thumb have three degrees of freedom (two pitchand one yaw); each of the two grasping fingers has one degree of freedom(pitch), and the palm member has one degree of freedom (pitch). Thewrist section is capable of moving palm housing 16 in directions havingtwo degrees of freedom (pitch and yaw) relative to the forearm sectionwhich houses the motors or actuators, and the circuitry and driveelectronics. The robotic hand, wrist section 12, and forearm section 14are individually discussed in further detail below, followed by adiscussion of a drive train, a lead screw assembly, the palm, a graspingfinger, a dexterous finger, and the thumb. More details on the robotichand is shown in U.S. Pat. No. 6,244,644, the content of which isincorporated by reference.

In one simple example, a user is on the way home and the system or theuser directly issues a command to prepare pizza for dinner. The hand canpick an microwaveable pizza from the refrigerator frozen section aheadof time to thaw the pizza, and then move the pizza to the microwaveoven, with instructions to coordinate the completion by the time theowner gets home, after appropriately considering traffic delays.Meanwhile, the dexterous hand can make salad and sprinkle a few freshlycut olives on the pizza when it is cooked.

In one embodiment, a cook top is positioned above the oven and includesa number of gas burners. Each of the burners has a grate positionedabove it, and the grates define a cooking surface. Each of the burnersis configured to produce a controlled flame that generates a quantity ofheat, which may be used to heat cooking utensils (i.e., pots and pans)placed on the grates. The burners and grates are arranged on the cooktopsuch that a user can simultaneously heat pots, pans, skillets, and thelike. The cooker has one or more cameras that monitor the cook topsurface for over boiling events or signs that the meat is overcooked(turning black/smoke) and reduces or cuts off the flame automatically.In addition, sensors 90 are used to determine when food is ready. Sensorcooking monitors the temperature and amount of steam coming from thefood to judge how much water remains and how long it should continueheating. In one embodiment, a process can estimate if the interior ofthe meat is done by monitoring temperature at the outside of the meat,the length of time in the oven, the color of the meat, the moisture inthe oven, among others. In another embodiment, gas sensors can react tospecific molecules. Multiple gas sensors can be used in an array,generally referenced as an “electronic nose”, so that more uniquepatterns can be detected in sampling readings across the multiplesensors.

A process can run the electronic nose recognition with one or moreclassifiers to detect the effect of time and distance on the gassensors' ability to react to the gas molecules released in an odor. Inone embodiment, the system segments the time series data intofive-second frames which are used to detect potency and diffusionchanges in near real time, while avoiding the delayed feedback thatwould be caused by a larger window. Then, each sample is normalized. Thesystem extracts a plurality of statistical features from each sensor'values collected from the gas sensors over each five-second frame. Thesefeatures can include: min, max, mean, standard deviation, sum, variance,slope, and y-intercept.

The classifier and the gas sensors enable a fuller picture of what ishappening in a kitchen. In addition to the suite of other sensortechnologies (e.g., audio or video, temperature, moisture), gas sensorshave the added benefit of being able to provide information previouslyinvisible to many other kitchen tracking systems, providing the abilityto not only classify the type of food, but also to help with nutritionaltracking. An oven has a cooking chamber into which pans, sheets, orother cookware carrying food may be placed to be heated. The oven mayalso have a built in camera to monitor whether the turkey has turned toa predetermined color and alters the cooking temperature. Also, thecamera in the chamber 40 monitors the food being cooked to apredetermined temperature and/or meat color as specified by a recipe.The cooking chamber 40 includes a number of racks located therein. Adoor assembly (not shown) is hinged to the front of the housing andpermits access to the cooking chamber. A gas-fired bake burner 44 withits associated cover is located below the rack. The bake burner isconfigured to provide heat for baking or otherwise cooking food items inthe cooking chamber. A user may control the operation of the oven usinga control interface located on the upper panel. The burner controldevice includes an electronically controlled gas valve operable tocontrol the supply of gas to the gas burner. While the cooker/oven hasbeen described as gas oven, the same principle applies to electriccooker/oven. Similar principles apply to slow cookers and multi-cookers,toasters, countertop ovens, microwave ovens, waffle bakers, amongothers.

The blenders, stand mixers and food processors all have a processor anda wireless communication module and the above described sensors. Inaddition, they can have rheometer sensors. A rheometer is used tomeasure the way in which a liquid, suspension or slurry flows inresponse to applied forces. It is used for those fluids which cannot bedefined by a single value of viscosity and therefore require moreparameters to be set and measured than is the case for a viscometer. Itmeasures the rheology of the fluid. Rheometers that control the appliedshear stress or shear strain are called rotational or shear rheometers,whereas rheometers that apply extensional stress or extensional strainare extensional rheometers. With rheometers, the blenders and othermixers can mix with precision.

The coffee makers and tea makers can have the above sensors includingtemperature sensors. These appliances can be set to remotely turn on,applied one or more water temperature curves over a preset time periodto make coffee or tea with perfection. In addition, they can be linkedto the refrigerators to dispense milk or other suitable additions.

For example, the smart tea maker can follow the following recipe: thewater should be below boiling. This is because the amino acids (whichproduce the tea's flavour) dissolve at lower temperatures than tannin.Tea made with water at 100° C. will be more astringent and less sweet.The process stops the kettle before it reaches the rolling boil—whiteand green teas are best at about 70° C., black and oolong teas use wateraround 85° C. For herbal infusions use 100° C. water, and 90° C. forChamomile. The tea can be left in the hot water for a predeterminedperiod and then it is ready for consumption.

FIG. 13B shows one exemplary motorized appliance such as a blender thatis IoT enabled with WiFi and/or Bluetooth. Appliance body houses a motor50, a drive shaft 52, and a coupling 54, as depicted in FIG. 3B. Whilethe embodiments shown include a motor of overly sufficient power (toquickly blend materials), it is contemplated that any motor can beutilized. Motor 50 is engaged with shaft 52 so that motor 50 causesrotation of shaft 52. Shaft 52 is connected with coupling 54, thuscausing rotation of coupling 54. Motor 50 is controlled by anInternet-Of-Thing (IOT) microprocessor 55 using suitable pulse widthmodulation (PWM) as known in the art. The IoT processor 55 can be a TICC3200 Wi-Fi wireless microcontroller (MCU) that is Wi-Fi CERTIFIED™ atthe chip level by the Wi-Fi Alliance™ with USB interface to PC forCCS/IAR using FTDI USB drivers and flash update over the USB. A powersupply 56 is also provided for powering motor 50. In certainembodiments, power supply 56 is a plug and cord for use with a standardwall socket. However, it is contemplated that motor 50 can be powered byseveral different types of power supplies. For example, power supply 56can be a lithium or other rechargeable battery. Additionally, standardbatteries such as AA or AAA batteries can be utilized. To reduceoverheating from running the motors, the system provides fans andcooling structures aimed at preventing excess heat from building up andbecoming a fire hazard, while still allowing for sufficient powerblasting of the motor. The Appliance may include a switch 58 forselectively providing power to motor 50. The switch can located withinthreaded opening 20, and is actuated by forcing container 14 towardsblender base. However, it is contemplated that other switches can beused, for example, standard on-off switch as are known to those ofordinary skill in the art.

It is contemplated that a smart container can be used in conjunctionwith various caps/lids. For example, as shown in FIG. 13C, a shaker top60, a storage lid 62, and a drinking rim 64 are provided. Shaker top 60is a cap with a plurality of holes useful for dispelling grated cheeseand the like from container. Storage lid 62 is a cap that substantiallyseals container 14 useful for storing used materials in a refrigerator.Both of these caps engage threaded opening 20 so as to remain fixed tocontainer 14. Drinking rim 64 also engages threaded opening 20, so as tocover the threads. Drinking rim 64 is useful for drinking directly fromcontainer. It is also contemplated that blender base 12 can be used withother attachments than the standard containers. For example, as is alsoshown in FIG. 3B, it is contemplated to use blender base with a juicerattachment 66 or a standard blender pitcher 68. However, there existother possibilities of attachments. The blender of FIGS. 3A-3B may ormay not have a graphical display. In one embodiment, the blender isremotely controlled using a smart phone or table or remote computer. Inanother embodiment, the blender is has a large display that can displayrecipes. A touch screen can be provided so that the user can interactover the Internet using the blender display, for example. Alternatively,the blender can interact with control app on a smart phone acting as auser interface.

A smart cap or a smart container can be provided using the system ofFIG. 2A, for example. One embodiment embeds electronics providingintelligence directly in a smart container. Other embodiments convertconventional containers into smart devices using smart cap. Whilemonitoring of the smart cap 100 is continuous, transmission of contentinformation can be continuous, periodic or event-driven. Themicrocontroller 155 is programmed with the appropriate warning and usagelevels, as well as internal damage diagnostics and self-recoveryfeatures. The content information can take any form, including a simplewarning/emergency indication that the content quality or age isapproaching or exceeding specifications, respectively. The sensorincludes a camera and image processor to detect if unusual films areforming on the surface of the food. The sensor can include theelectronic nose and/or the electronic tongue sensor detailed above todetect unusual odor or rotting condition. Other sensors includetime-temperature sensors that track the history of the container,microbial growth sensors, leakage sensors, shock sensors, andpathogen/contaminant sensors. In one embodiment to detect pathogens, asensor detects quality of food by adsorption of volatile biomarkers onselected materials coated on the sensor. When the volatile compoundsinteract with the coating materials, the change can be sensed byinfrared light directed towards the sensor, and a refractive indexchange causes wavelength shift of the infrared light reflected by thesensor. The wavelength shift is used to extract information on thevolatile compound and their concentrations inside the container orpackage. In another embodiment, biosensors can be used that include,enzyme, antibody or antigen based biosensors; gene based sensors andwhole cell sensor. Enzyme-based biosensors are based on electrochemicaltransduction systems with glucose oxidase sensors for example.Conjugated polymer based biosensors can be used that rely on indirectdetection of the target analyte, usually a fluorescently labelledcompound for biomolecular macromolecules such as proteins. Fluorescentsensors using boronic acid as a ligand, in a non-enzymatic approach forthe detection of saccharides have found applications in microbialdetection, as polysaccharides are a component of the bacterial cellmembrane.

Adsorption sensors can be used. Many substances can adsorb enzymes andother biological materials on their surfaces for example alumina,charcoal, clay, cellulose, kaolin, silica gel, glass, collagen, carbonpellets and advanced material such as carbon nanotubes (CNTs). A simpleprocedure is when microbial cells are immobilized by simple absorptionby placing the cells on a porous cellulose membrane. Generating pastessuch as when enzymes or tissue are mixed with graphite powder and liquidparaffin. Entrapment sensors provide physical enclosure of biomoleculein a small space. Inert membranes have been used to provide closecontact between the biomaterial and transducer. Types of membranes usedinclude cellulose acetate (dialysis membrane); polycarbonate(Nucleopore), synthetic non-permselective material; Collagen, a naturalprotein; PTFE: polytetrafluoroethylene (trade name Teflon) and is asynthetic polymer selectively permeable to gases. Nafion, (a Dupontmaterial), which is biocompatible and shown to be stable in cell cultureand the human body. Polymeric gels can be used and prepared in asolution containing the biomaterial. Chemical polymers such as calciumalginate, carrageenan, polyacrylamide, and sol-gel (Sol-gel, is a glassysilica produced by polymerization of silicate monomers). Bonding andcross linking: a number of bonding mechanisms have been used includingcovalent bonding A covalent bond exists between two atoms if they shareelectrons between them. The Biotin-Avidin bond is one of the strongestknown non-covalent bonds. Avidin is a terameric protein that forms ahighly specific binding site for Biotin. Sulphur compounds are known fortheir reactivity to metals and this absorb readily to the noble metals.Thiolised DNA can be attached to gold via different methods. Transducingelement: the transducing element must produce a measurable signal thatis proportionate to the concentration of the analyte/bioreceptor.Transducers can be divided into optical, electrochemical and mass based.

Optical transducers can be subdivided into light absorption,fluorescence/phosphorescence, reflectance, refractive index,bio/chemiluminiscence. In reflectance three widely used methods areSurface Plasmon resonance (SPR), total internal reflection fluorescence(TIFR) and attenuated total reflectance (ATR). Fiber optic biosensorscan be used in food matrixes to detect pathogens. Electrochemicaltransduction methods can be subdivided based on the measured parameter:amperometric (current), potentiometric (potential), impedimetric(impedance) and conductometric. Mass sensitive biosensors are suitablefor very sensitive detection, in which the transduction is based ondetecting a small changes in mass. The two main types of mass basedsensors are (1) bulk wave (BW) or quartz crystal microbalance (QCM) and(2) surface acoustic wave (SAW). The sensor 112-114, transceiver160/antenna 170, and microcontroller 155 are powered by and suitablepower source, which may optionally include an electromagnetic field(EMF) scavenging device 145, such as those known in the art, thatconvert ambient EMF (such as that emitted by radio station broadcasts)into small amounts of electrical power. The EMF scavenging device 145includes a battery to buffer and store energy for the microcontroller155, sensor 112-114, camera 140 and wireless communications 160/170,among others.

Food quality and other information from the microcontroller 155 arepreferably transmitted wirelessly through a wireless communicationmodule 160 and antenna 170. As stated above, the wireless communicationcomponent can use standard or proprietary communication protocols. Smartlids 100 can also communicate with each other to relay information aboutthe current status of the structure or machine and the smartcap/container themselves. In each smart container/smart cap, thetransmission of this information may be scheduled to be transmittedperiodically. The smart lid 100 has a data storage medium (memory) tostore data and internal status information, such as power levels, whilethe communication component is in an OFF state between transmissionperiods. On the other hand, once the communication commences in the ONstate, the microcontroller 155 can execute the following tasks: 1.Neighbor discovery: in this task each smart container or cap sends abeacon identifying its location, capabilities (e.g. residual energy),status. 2. Cluster formation: cluster head will be elected based on thefindings in (1). The cluster children communicate directly with theircluster head (CH). 3. Route discovery: this task interconnects theelected cluster heads together and finds the route towards the sinksmart container (node) so that minimum energy is consumed. 4. Datatransmission: the microcontroller processes the collected color data andbased on the adopted data dissemination approach, the smart cap will doone of the following. (a) Transmit the data as is without consideringthe previous status; or (b) transmit the data considering the previousstatus. Here we can have several scenarios, which include: (i)transmitting the data if the change in reported tension exceeds thewarning or emergency levels; and (ii) otherwise, do not transmit.

One embodiment of a smart container includes sensors and wirelesscommunication. Upon receiving a query over a wireless network, thecontainer can transmit the remaining volume, the expiration date, andorigin of the content. The expiration date and origin of the content canbe set at the factory, and the travel history of the container can beadded by the warehouse and the retailer. The bottles can have a levelsensor and a quality sensor to provide remaining volume and quality.Upon a radio frequency (RF) query by a processor, each container wouldrespond with identifying information, along with remaining amount and aquality rating, among others.

In a smart liquid container embodiment, a level sensor and a contentquality sensor can be embedded therein. For example, the level sensorcan determine remaining milk or juice in the container and respond to aquery. A MEMS based resistivity sensor can be placed in the container todetect if, for example, milk has gone sour. Alternatively, an electronicnose can be used to determine if the content is bad. In anothervariation, the lid of the container can include a camera to look formildew, among others. For moveable containers,

In one embodiment, a container for storing vegetable includes a camerawith image processing software to detect if mildew is present and if soan alert is generated. For leafy vegetables, the camera can detect ifthe leaves are wilting and also generate an alert. If frost is capturedby the camera, the container can send a request to reduce the vegetablebin temperature.

FIG. 14A shows an exemplary smart refrigerator 200 that stores the smartcontainers therein. In one embodiment, the refrigerator includes a realwindow or a virtual window view of the interior of the refrigerator. Thevirtual view is done with a camera that shows the content inside therefrigerator on a display 202 that is on a refrigerator exterior. Thisallows users to inspect the content of the refrigerator without openingthe refrigerator.

In a smart refrigerator embodiment, the smart containers can selfidentify upon query, so a user can ask containers with a particularexpiration date range to blink an LED associated with the smartcontainer. One embodiment uses a natural language interface coupled withspeech recognition. Various implementations can be used with Apple Siri,Android Voice, or Amazon Echo. For example, Amazon Echo is a hands-freespeaker controlled by voice. Echo connects to the Alexa Voice Service toplay music, provide information, news, sports scores, weather, amongothers. Echo has seven microphones and beam forming technology so it canhear voice commands from across the room—even while music is playing.When a user wishes to use voice command, the user can say the wake wordsuch as “Refrigerator”, “Oven” or “Alexa” (for Amazon Echo) and then acommand. The appliance would provide context to improve understanding ofthe command. For example, a speech enabled refrigerator can parse theverbal command “Refrigerator, identify expiring items” and issue acommand to all containers inside the refrigerator that meet theexpiration limit. The refrigerator can display the result on a displayoutside the refrigerator. The refrigerator can also display an interiorcam view of the frig, annotated with the location of matching itemsresponsive to a verbal query. The refrigerator can also identify lowinventory and suggests or adds to a shopping list.

One embodiment provides automatic inventory refill requests to asupermarket, Walmart, or Costco. In this embodiment, the refrigeratorbroadcasts a self-identification request to all containers inside therefrigerator. The containers reply with the ID and remaining quantity.Based on historical usage and user input on desired food, therefrigerator generates a refill order to a remote computer such as cloudbased inventory monitoring application 222.

FIG. 14B shows an exemplary process for on-line refrigerator inventoryreplenishment. The process is as follows:

Refrigerators in a local area provide real-time demand estimate (240)

Farmers predict cycle's harvest (242)

Refrigerator generates forecast of weekly demands (244)

System matches harvest cycle to each refrigerator requirement (246)

System automatically emails orders to the farmers and producers, whoharvest or prepare accordingly (248).

Goods are delivered from farmers or vendors to a local staging area orwarehouse and then packaged for delivery (250)

Owners either pickup from the local staging area or a ride-sharingservice can pick up and deliver in the same day (252)

Refrigerator inventory is updated (254)

In one embodiment, local suppliers such as farmers input predictions forthat cycle's harvest and the inventory is updated constantly during eachorder cycle to account for changes in the field. The refrigeratorsautomatically log on choose pickup locations, and shop from traditionalgrocery categories. Periodically, the system automatically emails ordersto the farmers and producers, who harvest or prepare according.Independently contracted drivers deliver the goods from vendors to thelocal warehouse. Foods are wrapped in compostable protective materials,and refrigerated items are insulated in inflated cool sleeves withbiodegradable ice packs. Everything is placed in insulated containersfor easier stacking, and each step is tracked for accountability.

Referring to FIG. 14C, a smart washer embodiment is shown with a cabinet412 with a front portion 415 and a rear portion 417. The front portion145 has an opening 430 closeable by a door 416. The clothes washerdescribed herein shares many features of a well-known clothes washer,and will not be described in detail except as necessary for a completeunderstanding of the invention. The cabinet 412 encloses a perforaterotatable basket 418 within a stationary imperforate tub 420. Clothingcan be thrown into the washer through the door 416. The cabinet 412 alsomounts a control panel 414 having control elements, such as switches,dials, buttons, and the like, operably coupled with a solid-statemicroprocessor-based controller 422 with an accelerometer and a camerafor controlling the operation of the clothes washer. The controller 422also interacts with a display panel 456 to provide additional userinterface with the user. The controller 422 can also communicate with asmart phone to receive instructions from the user, or can interact withthe internet through a WiFi or Zigbee connection, among others. Thecamera captures images of the clothing being washed and theprocessor/controller performs imaging functions that identify color,size of load, evenness of the load, and washing instructions imprintedon the clothing.

In one embodiment, prior to washing, a camera robot arm 438 snakesaround the clothing items and performs text recognition of the washinginstructions imprinted on washing labels. One embodiment of the roboticsnake arm is a flexible robotic limb which can function as a roboticsnake, here called a robo-snake. These improvements are particularlypertinent to miniaturization applications such as catheters orpositioners for microsurgery, micro-assembly, micro-manipulation, ormicro-exploration. This invention features improvements in a prior-artflexible robotic arm of Rennex, U.S. Pat. No. 5,386,741, the content ofwhich is incorporated by reference. The flexible arm has a series ofexpansible base units which were interconnected by six independentlycontrolled length actuators. This interconnection was accomplished withuniversal joints. This structure was very versatile is terms of itsmotion. Each stage could extend, tilt, twist, and expand or contractradially. The combination of stages could position its working end alonga tortuous path, and it could self-propel itself along a grid, a tunnel,or a blood vessel. It could also grip objects or position tools orimaging devices. Other features included optimal simplicity of control,ease of construction, lightness, and stiffness. The camera or anultrasonic cleaner such as an ultrasonic toothbrush can be mounted atone end of the robotic snake.

With the camera, the robot arm 438 also identifies color, and classifiesthe item such as shirt, pant, blanket, among others. The processor 422then segregates the items using the robot arm if possible, and otherwisewarns the user to manually separate the items to optimize the washingoperation.

IN one embodiment, the camera efficiently, reliably and accuratelysenses load size, the existence and magnitude of any imbalancecondition, and sense other obstructions that may adversely affectwashing performance and provides such information to the processor 422to actuate the motors to avoid imbalance conditions. The processor 422determines the mass of the vibrating system, including the basket 18,the tub 20, the axle 50, and the like, are readily determined. The massof the total clothes load can be determined in a well-known manner, suchas by evaluating the motor speed, current, or watts draw during motorstart up at a preselected time during the operation of the clotheswasher. Some methods require that the basket angular velocity passthrough the critical speed. Using the well-known relationship betweencentrifugal force caused by imbalanceF=mω2

where

F=centrifugal force caused by imbalance;

m=mass of imbalance;

R=radial location of the imbalance to rotating axis; and

w=angular velocity.

The moment M acting on the forward bearing can be calculated as

M=Fd,

where

d=longitudinal distance between imbalance load and forward bearing, and

M=moment acting on forward bearing.

The X-directional acceleration is directly proportional to the moment,M. When using an accelerometer according to the disclosed embodiment,the magnitude of the voltage signal from the accelerometer isproportional to the moment. The processor 422 can adjust the motor powerusing fuzzy logic to prevent the imbalance condition.

At times, the clothes washer may be supported upon a soft floor. At 140RPM a soft floor will cause increased vibration and potentiallyunacceptable vibration and cabinet hits at higher speeds. Theaccelerometer-based system can detect these increased vibrations andadjust the spin cycle accordingly. Similarly, improper installation ofthe clothes washer may result in the clothes washer being supported ononly three legs. This can also lead to increased vibration and cabinethits, which the accelerometer can detect at the 140 RPM speed, and thespin cycle can be adjusted accordingly.

The accelerometer-based system can also accommodate a load consisting ofa single bath towel or similar small, but readily imbalanced, load.Furthermore, in response to a load imbalance detected at a particularspeed, prior art technologies reduce the spin speed to the prior stagespin speed, which may be a 200 RPM decrease. With theaccelerometer-based system described herein, the spin speed is reduced50 RPM, thereby providing more effective extraction of liquidnotwithstanding the imbalance of the load.

One embodiment detects severe stains on the clothing, and performs spotcleaning prior to general washing by stretching the fabrics (in somecases over a hard surface) to increase their ability to attractcavitation bubble implosions and applying an ultrasonic applicator suchas an ultrasonic toothbrush to the stain areas. In the case ofstretching the fabric over a hard surface, the surface would actuallyact as the “receiver” of ultrasonic cavitation bubble implosions. Thefabric is in the intense cavitation zone adjacent to the hard surface.

The user can interact with the washer using the mobile phone. Forexample, the user can accept the system's segregation recommendation oroverride the recommendation. The user can remotely guide the washingmachine to operate immediately or run overnight to save electricitycosts for regions with time-based differential electricity pricing.Substantially the same structure exists in a dryer embodiment, but witha heater to dry the clothing rather than a water pump. In the dryerembodiment, the processor can cause the snaking robot to read dryinginstruction label text or icons.

FIGS. 14D-G show exemplary embodiments of refrigerator, water heater,clothes washer, and dish washer. FIG. 14D shows an exemplaryrefrigerator. The illustrated refrigerator 140 includes controlcircuitry 30 i (embodying a thermostat 142), a heating element 144, afan 146, a compressor 148, and a solenoid valve 150 in the depictedembodiment. Control circuitry 30 i, heater 144, fan 146, and compressor148 comprise exemplary loads 50 i in the depicted example. Therefrigerator 140 also include an ice energy storage chamber 152. Theembodiment uses ice storage to store coolth and used when the DR periodis active. First exemplary power management operations of controlcircuitry 30 i include adjustment of a temperature set point ofthermostat 142. It may be desired in at least one embodiment to set arelatively short duration of any temperature adjustment during powerarrangement operations. Another possible power management operationprovides temporary disablement of defrost operations of heating element144 (e.g., coupled with unillustrated coils of refrigerator 140), oradjusting a time of the defrost operations controlled by controlcircuitry 30 i. In another arrangement, heating element 144 may be usedto provide anti-sweat operations (e.g., appropriately positionedadjacent an exterior portion of an unillustrated cabinet of refrigerator140—for example adjacent to a door) and power management operations mayinclude temporary disablement of the anti-sweat operations or otherwiseadjusting such operations to occur at another moment in time whereinpower management operations are not being implemented. Additionalexemplary power management operations include disablement of interiorair circulation operations implemented by fan 146 and/or controllingoperations of compressor 148 (e.g., including temporarily disabling orreducing the speed of compressor 148). Additional aspects includeimplementing a hot gas bypass operation of compressor 148 using solenoidvalve 150 and as described in further detail above in one example. Oneother embodiment provides a multi-stage refrigerator 140 having aplurality of cooling stages and a power management operation includescontrolling the refrigerator 140 to operate at less than the availablenumber of cooling stages thereby reducing the amount of energy consumedby the appliance.

In one implementation, the refrigerator ice energy storage chamberprovides a predetermined cold energy for a refrigerated volume for thepredicted DR period; and a fan to circulate cold air from the ice energystorage chamber inside the refrigerated volume during the predicted DRperiod. The refrigerator can include a phase change material (such aswater, salted water, paraffin, among others) coupled to the refrigeratedvolume to maintain the refrigerated volume at a predeterminedtemperature during the predicted DR period. The controller modulatescompressor operation to reduce power consumption during the predicted DRperiod. The controller precharges the refrigerator prior to thepredicted DR period. The controller precharges the refrigerator based onweather or warning from an authority. The refrigerator can include icestorage to store coolth when power is available and used during the DRperiod. The fact that water is a pure substance and that making ice doesnot involve a chemical reaction is one reason that ice storage is arelatively trouble free system.

The phase change materials include alkanes, paraffin waxes and salthydrates. These materials undergo a reversible solid to liquid phasechange at various transition temperatures. ‘Solid-state’ phase changematerials are those that change from amorphous to crystalline phaseswhile remaining ‘solid.’ Both paraffin wax and salt hydrates typicallyrequire encapsulation to contain the liquid phase, which adds to finalcost of this PCM. Salt hydrates are inorganic materials. Inorganiccompounds have twice the volumetric latent energy storage compared toorganic compounds. The organic compounds however, have the advantages ofmelting congruently and are non-corrosive. Salt hydrates will meltincongruently causing phase separation. There are two categories ofsolid-state phase change materials: layered perovskites and plasticcrystals. The transition temperature of solid-state phase changematerials in a pure form runs on the higher side for use in passiveapplications. By mixing these compounds in various ratios, thetransition temperature can be lowered.

PCM can use paraffin waxes which are part of a family of saturatedhydrocarbons. The structure is the type C n H 2n+2. Those with carbonatoms between five and fifteen are liquids at room temperatures and arenot considered. Normal or straight chain and symmetrically branchedchain paraffin waxes are the most stable. Typically, paraffin waxes withodd numbers of carbon atoms are more widely used because they are moreavailable, more economical and have higher heats of fusion. Paraffinwaxes are composed mainly of alkanes, approximately 75%. Alkanes andparaffin waxes are both organic compounds. Paraffin can contain severalalkanes resulting in a melting range rather than a melting point. As themolecular weight increases, the melting point tends to increase as well.Using different mixtures of alkanes, specific transition temperaturesfor paraffin waxes can be attained. Paraffin waxes and alkanes at thetransition temperature melt to a liquid and solidify upon cooling. Theydo not have the containment problems of salt hydrates. The properties ofnormal paraffin wax are very suitable for latent heat storage. They havea large heat of fusion per unit weight, they are non-corrosive,nontoxic, chemically inert and stable below 500° C. (932° F.). Onmelting, they have a low volume change and a low vapor pressure. Mixingdifferent molecular weight paraffin waxes together can easily varymelting temperature. Since they are commercially available, the cost isreasonable. Prime candidates for passive applications are tetradecane,hexadecane, octadecane and eicosane. Paraffin wax has a low thermalconductivity. However, the addition of additives such as graphite couldincrease the thermal conductivity. A Boulder, Colo. company, OutlastTechnology, distributes outerwear made of fabrics that incorporateencapsulated paraffin wax. The Outlast Technology fabric involves themicroencapsulation of microscopic size droplets of paraffin wax. Theseencapsulated particles of wax are then incorporated into fabrics andfoams that are used for lining materials.

Octadecane in its pure form has a relatively high heat of fusion with atransition temperature close to an ideal passive temperature. Its latentheat storage is more than three times greater than the NPG/PG mixture.Based on thermal storage capabilities, octadecane is the superiormaterial, followed by the Kenwax 18.

Paraffin wax and solid-state phase change materials show the behavior ofunder or super cooling. This behavior occurs when the material does notsolidify at the same temperature at which it melted. Solid-state phasechange materials have shown more than a twenty-degree difference. Thedifference is not as noticeable in paraffin waxes. Other phase changematerials can be used.

FIG. 14E shows an exemplary water heater. Water heater 150 includescontrol circuitry 30 h (embodying a thermostat 152 in the illustratedconfiguration) and a heating element 154. Heating element 154 isconfigured to heat water in a main reservoir 156 and an associatedreservoir 158 to a desired temperature in the depicted configuration.Control circuitry 30 h and heating element 154 comprise loads 50 h ofwater heater 150 in one embodiment.

According to an illustrative embodiment, power management operations ofsystem 150 and implemented by control circuitry 30 h include adjusting aset point of thermostat 152. For example, the thermostat set point maybe temporarily lowered (e.g., for a period of tens of seconds, or a fewminutes in some examples). In other exemplary power managementoperations, control circuitry 30 h may directly disable or provide othercontrol of heating element 154 and gate pre-heated water from the backup reservoir 158 during the DR period.

According to additional exemplary aspects, a set point of any of thethermostats disclosed herein of the various appliances may be assignedto one of a plurality of possible power management set points accordingto a monitored condition of electrical energy of system 101. Forexample, a scale of set points may be used according to the condition ofthe electrical energy (e.g., the temperature set point may be decreasedat predefined decrements (1-10 degrees for example) corresponding to thesystem frequency of the electrical energy deviating respectivepredetermined amounts (e.g., 10 mHz) from the nominal frequency. Inaccordance with the described example, the magnitude of adjustment ofthe thermostat set point increases as the deviation of the systemfrequency from the nominal frequency increases.

In one implementation, the water heater 150 uses the back-up heatedenergy storage chamber 158 to store a reserve heated water to maintaineda predetermined temperature output for the water heater during thepredicted DR period; and a valve to mix the reserve heated water withthe water in the main water heater tank during the predicted DR period.The water heater can include a phase change material coupled to thewater volume to maintain the water at a predetermined temperature duringthe predicted DR period. The water heater controller modulates heateroperation to reduce power consumption during the predicted DR period.The controller can precharge the water heater prior to the predicted DRperiod or based on weather or warning from an authority.

FIG. 14F shows an exemplary clothes washer. In one implementation, thewasher has a digitally actuated latch to secure a washer door during thepredicted DR period. If the washer is in an uninterruptible cleaningoperation during the predicted DR period, the controller reduces powerconsumption during the predicted DR period and subsequently repeats theuninterruptible operation after the predicted DR period. Further, if thewasher is in an extendible cleaning operation during the predicted DRperiod, the controller reduces power consumption during the predicted DRperiod and subsequently completes the extendible operation after thepredicted DR period. The washer appliance can include a back-up heatedenergy storage chamber to store a reserve heated water to maintained apredetermined temperature output for the heater during the predicted DRperiod; and a mixer to mix the reserve heated water with cold water tomaintain a predetermined washing temperature during the predicted DRperiod. A data input device can indicate the use of detergent additiveor bleach usage, wherein the processor ignores the predicted DR periodto avoid damage to items in the washer. The exemplary clothes washer 160may include control circuitry 30 d, a heating element 162, and anagitator motor 164. Heating element 162 is configured to heat water usedin an associated compartment (not shown) of clothes washer 160configured to receive and wash clothes. Heating element 162 is also usedto heat a water reservoir 168 for use during the temporary DR period sothat washing operations can continue. Agitator motor 164 is configuredto oscillate between different rotational directions or otherwiseagitate clothes within the associated compartment during wash and/orrinse operations. Control circuitry 30 g, heating element 162 andagitator motor 164 comprise associated loads 50 g of clothes washer 160in the depicted embodiment. In one configuration, power managementoperations of clothes washer 160 include reducing or ceasing the supplyof electrical energy to heating element 162 to reduce internaltemperatures of water in the associated compartment and/or agitatormotor 164 to reduce motion of the motor 164. The reduction in power bycontrolling heating element 162 may be linear and accordingly thebenefits may be directly proportional to the reduction in the watertemperature. The reduction in power to agitator motor 164 may beproportional to a product of angular acceleration, mass and angularvelocity. A slowing down of agitator motion of motor 164 could affectboth a reduction in acceleration as the motor reverses its motion aswell as angular velocity. In other embodiments, it may be desired tomaintain agitator motor 164 in an operative mode during animplementation of power management operations with respect to heatingelement 162.

An exemplary clothes dryer may similarly include control circuitry, aheating element, and a tumbler motor. Heating element is configured inone embodiment to heat an associated compartment (not shown) of clothesdryer configured to receive and dry clothes. Tumbler motor is configuredto spin clothes within the associated compartment during dryingoperations. In one configuration, power management operations of clothesdryer include reducing or ceasing the supply of electrical energy toheating element (e.g., reducing an amount of current supplied to heatingelement) and/or tumbler motor. It may be desired to maintain tumblermotor in an operative mode during an implementation of power managementoperations with respect to heating element.

FIG. 14G shows an exemplary dish washer. Dish washer 170 includescontrol circuitry 30 f, a water heating element 172, a forced airheating element 174, and a water pump 176 in but one embodiment. Dishwasher 170 may additionally include a compartment (not shown) configuredto receive to dishes. Water heating element 172 may adjust a temperatureof water used to wash dishes using dish washer 170 in one embodiment.Heating element 172 is also used to heat a reservoir 178 to provide hotwashing water during a DR period. Forced air heating element 174 adjustsa temperature of air used to dry the dishes in one implementation. Waterpump 176 may spray water on the dishes during a cleaning and/or rinsingcycle to provide a dish cleaning action and/or rinsing action. Controlcircuitry 30 f, heating elements 172, 174, and water pump 176 maycomprise associated loads 50 f of dish washer 170.

Exemplary power management operations of dish washer 170 implemented bycontrol circuitry 30 f in one embodiment include controlling the waterheater 172 to reduce a water temperature boost cycle during washoperations and/or reduce air temperature by forced air heater 174 duringrinsing/drying operations. Reduction of water temperature providescorresponding linear reductions in electrical power consumption. Controlcircuitry 30 f may also control operations of water pump 176 (e.g.,reduce the operational speed of pump 176) during modes of reduced powerconsumption.

The IOT appliances include one or more of the following aspects:

1. A kitchen system, including:

a body; a processor in the body and coupled to a wireless transceiver; acamera coupled to the body; and an accelerometer to detect acceleration;

an oven appliance or a cooker appliance coupled to a network;

a refrigerator appliance coupled to the network;

a robotic arm to move one or more containers between appliances for foodcooking or drink mixing; and

a smart phone, watch, tablet, or mobile computer wirelessly coupled toone or more of the appliances to control settings on the appliance.

2. The kitchen system, wherein the appliance comprises:

a processor;

-   -   one or more sensors coupled to the processor; and    -   code to receive a remote control from the smart phone, watch,        tablet, or mobile computer and to actuate a valve, motor,        sensor, or actuator in response to the remote control.

3. The kitchen system, wherein at least one appliance comprises acamera.

4. The kitchen system, wherein the appliance monitors inventory andcommunicates a reorder request to a remote computer.

5. The kitchen system, wherein the appliance communicates usage, fancomments, or cooking information from a brand to a user.

6. The kitchen system, wherein the appliance complies with a request ordemand from a utility to reduce power consumption for a period.

7. The kitchen system, wherein the appliance comprises one or moresensors to sense progress of food cooking or drink mixing.

8. The kitchen system, wherein the appliance detects a potentialcomponent failure and requests service prior to a component failure.

9. The kitchen system, including a robot to move items between at leasttwo appliances.

10. The kitchen system, wherein each appliance comprises anInternet-of-Things (IOT) appliance.

11. The kitchen system, including a 3D printer to form food into apredetermined 3D shape.

12. The kitchen system, including a syringe moveable in 3D, the syringereceiving food particles and having a computer controlled plunger todispense the food particles to form a 3D shape.

13. The kitchen system, including a plurality of syringes each moveablein 3D, each syringe receiving food particles with a computer controlledplunger to dispense the food particles to form a 3D shape.

14. The kitchen system, wherein the food particles from each syringe aremixed together.

Smart Refrigerator

1. A refrigerator, including:

-   -   a body; a processor in the body and coupled to a wireless        transceiver; a camera coupled to the body; and an accelerometer        to detect acceleration;    -   a plurality of shelves or bins that receive digitally responsive        containers requiring refrigeration;    -   one or more wireless charging pads or layers each positioned on        a shelf to power the digitally responsive containers;    -   a transceiver coupled to a network to access the Internet and to        communicate with one or more appliances coupled to the network;        and    -   a smart phone, watch, tablet, or mobile computer wirelessly        coupled to the refrigerator to control refrigerator settings or        receive status of the digitally responsive containers.

2. The refrigerator, including a camera to detect spoilage or positionof a container.

3. The refrigerator, wherein the container responds to a query includingremaining amount and freshness.

4. The refrigerator, including a camera coupled to the processor and oneor more volume indicia on the container to indicate remaining content.

5. The refrigerator of claim 2, including mildew detection code todetermine the presence of mildew in a container.

6. The refrigerator, including a wireless power transmitter coupled to acontainer to transfer power from a layer or sheet of wireless powertransmitters and wherein the layer or sheet of wireless powertransmitters receive container status data.

7. The refrigerator, including code to report content status, age,quality, or how to enjoy the content.

8. The refrigerator, including code to contact a consumer, distributor,retailer, or manufacturer with content status.

9. The refrigerator, including an electronic nose to detect an odorindicating content quality.

10. The refrigerator, including code to sequence usage of the containerin a first-in-first-out (FIFO) order to reduce spoilage.

11. The refrigerator, including code to determine if the content isbelow a predetermined threshold and if so order additional supply.

12. The refrigerator, including a 3D ice printer to form a predetermined3D ice shape.

13. The refrigerator, including a syringe moveable in 3D, the syringereceiving liquid and having a computer controlled plunger tosequentially dispense the liquid in a layer and to freeze the layer toform a 3D shape.

14. The refrigerator, including a plurality of syringes each moveable in3D, each syringe receiving liquid with a computer controlled plunger todispense the liquid layer.

15. The oven, wherein the liquid particles from each syringe are mixedtogether.

16. The refrigerator, including code to communicate usage, fan comments,or cooking information from a brand to a user.

17. The refrigerator, including code to comply with a request or demandfrom a utility to reduce power consumption for a period.

18. The kitchen system, including a robotic arm to move the container toan appliance for food cooking or drink mixing.

19. The refrigerator, including code to detect a potential componentfailure and requests service prior to a component failure.

20. A system, including:

a body; a processor in the body and coupled to a wireless transceiver; acamera coupled to the body; and an accelerometer to detect acceleration;

an oven or a cooker coupled to a network;

a refrigerator coupled to the oven or cooker appliance, including:

-   -   a plurality of shelves or bins that receive digitally responsive        containers requiring refrigeration;    -   one or more wireless charging pads or layers each positioned on        a shelf to power the digitally responsive containers;    -   a transceiver coupled to a network to access the Internet and to        communicate with one or more appliances coupled to the network;

a robotic arm or conveyor to move an item from the refrigerator to theoven or cooker; and

-   -   a smart phone, watch, tablet, or mobile computer wirelessly        coupled to the refrigerator to control refrigerator settings or        receive status of the digitally responsive containers.

Smart Container

1. A container to store content therein, including:

-   -   a body; a processor in the body and coupled to a wireless        transceiver; a camera coupled to the body; and an accelerometer        to detect acceleration;    -   a volume sensor for the content;    -   a wireless transceiver; and    -   a processor coupled to the wireless transceiver and the volume        sensor, the processor responding to a query on remaining content        and quality of the content.

2. The container, including a camera coupled to the processor and one ormore volume indicia on the container to indicate remaining content.

3. The container including mildew detection code to determine thepresence of mildew in the container.

4. The container, including rechargeable battery or super-capacitorcoupled to a wireless power receiver in the container to receive powerfrom a layer or sheet of wireless power transmitters.

5. The container wherein the wireless power receiver comprises an NFCreceiver and wherein the layer or sheet of wireless power transmittersreceive container status data.

6. The container, including code to report status including content ageor quality.

7. The container, including code to suggest best ways to enjoy thecontent.

8. The container, including an electronic nose to detect an odorindicating content quality.

9. The container, including code to contact a consumer, distributor,retailer, or manufacturer with a status of the content.

10. The container including code to exchange information between theconsumer and a distributor, retailer or manufacturer of the content.

11. The container, including a sensor to detect container opening.

12. The container, including an anti-fraud lid and registration code todetect product tampering or product fraud.

13. The container, including a MEMS sensor to detect liquid quality,wherein the MEMS sensor detect liquid conductance or resistance andcompares with a predetermined quality rating.

14. The container, including code to sequence usage of the container ina first-in-first-out (FIFO) order to reduce spoilage.

15. The container, including code to determine if the content is below apredetermined threshold and if so ordering additional supply.

16. The container 5, including code to determine expiration date andsignaling a user to use prior to the expiration date.

17. A system, including:

an appliance;

a digitally responsive container to store content therein, the containerreceived by the appliance, the container, including:

-   -   a volume sensor for the content;    -   a wireless transceiver; and    -   a processor coupled to the wireless transceiver and the volume        sensor, the processor responding to a query on remaining content        and quality of the content; and

a smart phone, watch, tablet, or mobile computer wirelessly coupled toone or more of the appliances to control settings on the appliance.

18. The system 7, including a camera coupled to the processor and one ormore volume indicia on the container to indicate remaining content.

19. The system, wherein the container comprises mildew detection code todetermine the presence of mildew in the container.

20. The system, wherein the container comprises rechargeable battery orsuper-capacitor coupled to a wireless power receiver in the container toreceive power from a layer or sheet of wireless power transmitters.

Oven/Cooker

1. An apparatus to cook food, including:

-   -   a body; a processor in the body and coupled to a wireless        transceiver; a camera coupled to the body; and an accelerometer        to detect acceleration;    -   a cooking area to receive food, the area heated by a heater;    -   one or more sensors to detect food cooking parameters;    -   a camera to detect food color change;    -   a processor coupled to the camera and to the one or more        sensors;

a transceiver coupled to a network to access the Internet; and

-   -   a smart phone, watch, tablet, or mobile computer to control        settings or receive status of the apparatus.

2. The apparatus, including a temperature sensor to sense foodtemperature.

3. The apparatus, including a tag communicator in communication with atag on the food to access food content or cooking instruction.

4. The apparatus, including code for changing cooking modes for optimalcooking, wherein the cooking modes include temperature variations over apredetermined period.

5. The apparatus, including a tag with cooking data on a package,wherein the tag configures settings based on the cooking data.

6. The apparatus, wherein the tag comprises a bar code or an NFC code.

7. The apparatus, wherein the camera transmits food images duringcooking to a user.

8. The apparatus, wherein the camera automatically identifies the foodbeing cooked and switches heating settings.

9. The apparatus, including a scale to measure weight, wherein theweight is used to adjust cooking.

10. The apparatus, including a humidity sensor.

11. The apparatus, including a microwave heat source to cook food.

12. The apparatus, including a 3D printer to form food into apredetermined 3D shape.

13. The apparatus, including a syringe moveable in 3D, the syringereceiving food particles and having a computer controlled plunger todispense the food particles to form a 3D shape.

14. The apparatus, including a plurality of syringes each moveable in3D, each syringe receiving food particles with a computer controlledplunger to dispense the food particles to form a 3D shape.

15. The apparatus, wherein the food particles from each syringe aremixed together.

16. The apparatus, including a robotic assembly to move an item from arefrigerator into the cooking area.

17. The apparatus, including code to detect usage and recommend heaterreplacement.

18. The apparatus, wherein the cooking area comprises a microwave ovenchamber, slow cooker pot, a coffee pot, a waffle baker, or a toasterchamber.

19. A system to cook food, including:

a body; a processor in the body and coupled to a wireless transceiver; acamera coupled to the body; and an accelerometer to detect acceleration;

a refrigerator;

an cooking appliance, including:

a cooking area to receive food;

-   -   one or more sensors to detect food cooking parameters;    -   a camera to detect food color change;    -   a processor coupled to the camera and to the one or more        sensors;

a transceiver coupled to a network to access the Internet; and

a robotic arm or conveyor to move an item from the refrigerator to thecooking appliance based on a command; and

a smart phone, watch, tablet, or mobile computer wirelessly coupled tothe refrigerator and the oven to move the item from the refrigerator tothe oven and to control oven settings or receive status of the oven.

20. The system, wherein the cooking area comprises an oven chamber, slowcooker pot, a toaster chamber, microwave chamber, slow cooker pot,coffee pot, waffle baker, or a grill surface.

Smart Clothing Washer

1. A washer, including

-   -   a chamber or a body; a processor in the body and coupled to a        wireless transceiver; a camera coupled to the body; and an        accelerometer to detect acceleration;    -   a motor coupled to the chamber to wash an item;    -   a valve coupled to the chamber to control water flow into the        chamber;

a drain valve coupled to the chamber to remove water from the chamber;

-   -   a camera to inspect items in the chamber; and

a processor coupled to the camera, the processor running imageprocessing to identify a soiled region for additional cleaning.

2. The washer, including code to detect stains on the item.

3. The washer, wherein the washer comprises a clothing washer.

4. The washer, including a snaking robot arm to inspect an individualitem for stain or for washing instruction.

5. The washer, including an ultrasonic cleaning head coupled to thesnaking robot arm to preclean a soiled area.

6. The washer, including code to change a washing program based ondetected characteristics of a clothing item and a detergent.

7. The washer, including an accelerometer to detect chamber movement andcode to balance a chamber given a predetermined load during washing.

8. The washer, including code to segregate clothing items according tocolor, size, or washing instruction.

9. The washer, wherein the washer comprises a dishwasher.

10. The washer, wherein the camera detects dirtiness and quantity ofdishes and set the cleaning cycle therefrom.

10. The washer, wherein the camera detects a washing load with acasserole dish or bread pan and directs additional water at the washingload.

11. The washer including a robotic arm to move items from the washer toanother chamber.

12. The washer, including RFID tag in each item to provide informationabout the item type and washing instruction.

13. The washer, including code to perform diagnostics on the washer,change its status, upgrade firmware, and launch an app that communicatesdirectly with a manufacturer's service center.

14. The washer, including a voice recognizer to answer a question or tochange washing parameters.

15. The washer, including an actuator to transport clothing into adrying chamber.

16. The washer, including an actuator to transport clothing into adryer.

17. The washer, wherein the dryer includes:

-   -   a chamber;    -   a motor coupled to the chamber;    -   a heater to warm the chamber;

a humidity sensor to sense humidity in the chamber; and

a processor coupled to the humidity sensor to stop when a humidity inthe chamber reaches a threshold.

18. The washer, including an actuator to transport clothing into adrying chamber.

19. The washer, including a plurality of detergent nozzles each having avalve and an actuator to transport clothing into a drying chamber.

20. The dryer, including a smart phone, watch, tablet, or mobilecomputer wirelessly coupled to the dryer to control settings or receivestatus of items in the chamber.

Smart Clothing Dryer

1. A dryer, including

-   -   a chamber or a body; a processor in the body and coupled to a        wireless transceiver; a camera coupled to the body; and an        accelerometer to detect acceleration;    -   a motor coupled to the chamber;    -   a heater to warm the chamber;

a humidity sensor to sense humidity in the chamber; and

a processor coupled to the humidity sensor to stop when a humidity inthe chamber reaches a threshold.

2. The dryer, including a camera and code to detect fabric type or imageprocessing code to recognize drying instruction on an item.

3. The dryer, including an RF unit to communicate with an RF tag on anitem to access drying instruction on the item.

4. The dryer, including a robotic arm to segregate items based on dryingfactors.

5. The dryer, including a snaking robot arm to inspect an individualitem for drying factors.

6. The dryer including an ultrasonic cleaning head coupled to thesnaking robot arm to perform additional cleaning on a soiled area.

7. The dryer, including an accelerometer to detect chamber movement andcode to balance the chamber given a predetermined load during drying.

8. The dryer, including code to segregate clothing items according tocolor, size, or drying instruction.

9. The dryer, including code to change a drying program based ondetected characteristics of a clothing item and a fabric softener.

10. The dryer, including a sorter to separate clothing items into apredetermined drying period or temperature.

11. The dryer, including a robotic arm to move items from the dryer toanother chamber.

12. The dryer, including RFID tag in each item to provide informationabout the item type and drying instruction.

13. The dryer, including code to perform diagnostics on the dryer,change its status, upgrade firmware, and launch an app that communicatesdirectly with a manufacturer's service center.

14. The dryer, including a voice recognizer to answer a question or tochange washing parameters.

15. The dryer, including a smart phone, watch, tablet, or mobilecomputer wirelessly coupled to the dryer to control settings or receivestatus of items in the chamber.

16. The dryer, including an actuator to transport clothing into awashing chamber.

17. The dryer, including an actuator to transport clothing into awasher.

18. The dryer 7, including:

-   -   a motor coupled to the chamber to wash an item;    -   a valve coupled to the chamber to control water flow into the        chamber;

a drain valve coupled to the chamber to remove water from the chamber;

-   -   a camera to inspect items in the chamber; and

a processor coupled to the camera, the processor running imageprocessing to identify a soiled region for additional cleaning.

Motorized Cooking Appliance

1. An appliance, including:

-   -   a body; a processor in the body and coupled to a wireless        transceiver; a camera coupled to the body; and an accelerometer        to detect acceleration;    -   a surface with a sensor to sense characteristics of food or        drink;    -   a motor to mix the food or drink;    -   a wireless transceiver connected to the Internet; and    -   a processor coupled to the wireless transceiver and the sensor,        the processor running the motor according to a suggestion on the        Internet on preparing the food or drink.

2. The appliance, including a camera coupled to the processor to detectcondition of the food or drink.

3. A system, including:

an appliance with a body; a processor in the body and coupled to awireless transceiver; a camera coupled to the body; and an accelerometerto detect acceleration;

a digitally responsive appliance to store content therein, the containerreceived by the appliance, the container, including:

-   -   a volume sensor for the content;    -   a wireless transceiver; and    -   a processor coupled to the wireless transceiver and the volume        sensor, the processor responding to a query on remaining content        and quality of the content; and

a smart phone, watch, tablet, or mobile computer wirelessly coupled toone or more of the appliances to control settings on the appliance.

18. The system, including a camera coupled to the processor and one ormore volume indicia on the container to indicate remaining content.

19. The system, wherein the container comprises mildew detection code todetermine the presence of mildew in the container.

20. The system, wherein the container comprises rechargeable battery orsuper-capacitor coupled to a wireless power receiver in the container toreceive power from a layer or sheet of wireless power transmitters.

Voice Controlled Appliance

An IOT voice controlled assistant may be worn on the body (wearable) ormay be positioned in a room (e.g., at home, work, store, etc.) toreceive user input in the form of voice interactions, such as spokenrequests or a conversational dialogue. Depending on the request, thevoice controlled assistant may perform any number of actions. Forinstance, the assistant may play music or emit verbal answers to theuser. The assistant may alternatively function as a communication deviceto facilitate network voice communications with a far end talker. Asstill another alternative, the user may ask a question or submit asearch request to be performed by a remote cloud service. For instance,the user's voice input may be transmitted from the assistant over anetwork to the cloud service, where the voice input is interpreted andused to perform a function. In the event that the function creates aresponse, the cloud service transmits the response back over the networkto the assistant, where it may be audibly emitted.

In one embodiment, the portable voice controlled assistant 104 includesa wireless transceiver such as WiFi and/or Bluetooth for sending andreceiving authentication information to a computing device which can belocal or cloud-based. In one embodiment, the authentication informationis sent directly to the cloud and does not pass through the operatingsystem of the local processor. Hence, the authentication process isindependent of the operating system of the processor and any errors orsecurity failures in the operating system do not affect the security ofdata storage device 10. In the embodiment of FIG. 14H, the systemsecures voice controlled assistant 104 with the use of a mobile phone asthe wireless security key. The mobile phone uses a wireless transceiversuch as WiFi and/or Bluetooth for sending and receiving authenticationinformation to the voice controlled assistant. This communication issecured by encrypting the message using public key cryptography. Thesystem has two software components: a component on the mobile phone anda component on the voice controlled assistant. The component on thevoice controlled assistant is responsible for checking if a mobile phonewith the wireless security key is nearby, sending challenge message tothe mobile phone and receiving validation message from the mobile phone.If the wireless security key is not nearby, the software component willlog out from Windows system. Otherwise, it will log in or keep thelogging in state. The component on the mobile is responsible forsecurity validation. In one implementation, the medium is Bluetooth. Inthe Bluetooth implementation, the typical range is about 10 m and themaximum range is about 100 m, and the system can simply default to arange of 10 m before an alarm is generated by detecting when theBluetooth signal from the smart phone is lost. In anotherimplementation, the wireless link is 802.11 (WiFi). In the WiFiembodiment, the distance can be much greater than 100 m, and thesoftware can detect range by scaling the RSSI (received signal strengthindicator) flag from the WiFi transceiver chip so that the RSSI valuecorresponds to the predetermined range. In yet a third implementation,the wireless link is a combination of 802.11 and Bluetooth. After asecure authorization has already been obtained, for instance, theprocess maintains a secure authorization between the mobile device andthe voice controlled assistant. The process flow includes waiting untila predetermined period to elapse, reestablishing the secure channelbetween the wireless mobile device and the voice controlled assistant.The process determines if the authentication succeeds or fails, anddenying access to the voice controlled assistant if the authenticationfails.

Following the conclusion of the process flow of FIG. 14H, a secureauthorization has been established between the voice controlledassistant 104 and a mobile communication device 10. This authorizationmust be periodically refreshed to ensure that the voice controlledassistant 104 is still within the immediate vicinity of thecommunication device 10. Thus, operations to the voice controlledassistant 104 are permitted until a predetermined time has elapsed.After interval, the voice controlled assistant 104 reestablishes thesecure authenticated channel with the mobile device 10. If theauthentication succeeds the device returns to the authenticated stateand if not, the device goes to an unauthenticated state and will denyaccess to the voice controlled assistant 104.

FIG. 14I is another exemplary diagram of a simplified process flowshowing wireless communication between the mobile communication device10 and a voice controlled assistant 104 to establish a secureauthorization according to an embodiment of the present invention. Theprocess flow includes step 42 to pair up the mobile communication device10 and the voice controlled assistant 104. In the pairing operation, theprocess determines the unique ID (such as the processor ID of bothdevices 8 and 10). During operation, in step 44, the voice controlledassistant 104 powers up in a password authentication mode. In step 46,the process determines if the mobile communication device 10 and voicecontrolled assistant 104 are within wireless communication range andthat the paired IDs match. In 48, if not in range, the mobilecommunication device 10 can provide an audio beep or a vibration toindicate that the user has separated from the voice controlledassistant. In step 50, the process executes an authentication protocolbetween the mobile communication device and the voice controlledassistant, and to begin the secure session in the voice controlledassistant. In step 52, the voice controlled assistant 104 determines ifthe authentication protocol has been successful, if it has the processcontinues to step 54 and if not it continues to step 56. In step 56, thedevice increments a counter which specifies a period to wait and waitsthat period of time before returning to step 46. Optionally, a key todecrypt data on the storage device is sent from the mobile communicationdevice to the voice controlled assistant over the establishedauthenticated communications channel In step 54, the voice controlledassistant computer 8 is unlocked and allows operation of the computer.In another embodiment, the voice controlled assistant can use the keyprovided by the mobile device 10 to decrypt and encrypt data asrequired.

FIG. 14J shows an exemplary video based authentication. In 62, thesystem trains the voice assistant to authenticate user viapicture/video. In 64, the voice assistant powers up in a passwordauthentication mode and in 66 detects voice command and check securitylevel of the command. In 68, if the system encounters a secured request(such as turn off alarm or unlock a door), the system goes into the nextlevel of check by turning on a camera to capture and visuallyauthenticate the user. In 70 the system runs an authentication protocolbetween the mobile communication device and the voice assistant. In 72the system checks if the authentication protocol is successful. In 76,if unsuccessful, the device increments a counter which specifies aperiod to wait and waits that period of time before allowing anotherauthentication. In 78, if authenticated, voice assistant secure mode isunlocked and allows user full control of appliances by voice.

In one embodiment, the communication between the mobile device 10 andthe voice controlled assistant is secured. In addition to encrypting themessage using public key cryptography, the message may be additionallyprotected by using a digital certificate. A certificate authorityfunctions as a trusted party known to both the voice controlledassistant 104 and the cell phone 10. The certificate authority possessesboth a public and private key, of which the private key is closelyguarded. The public key of the mobile device 10 may be encrypted usingthe private key of the certificate authority. This constitutes a digitalcertificate that can be used to help authenticate different devices, inthis case the mobile device 10 and the voice controlled assistant 104 toeach other using the certificate authority. The certificate may bestored in a data storage device with the unique public and private keysof the voice controlled assistant.

In still another embodiment of the present invention, the electronics ormotor within the voice controlled assistant 104 will not functionwithout having established a secure authorization between the mobilecommunication device 10 and the voice controlled assistant 104. Powermay be temporarily suspended to components within the voice controlledassistant, or the motor may be prevented from operating until a secureauthorization was established.

In other embodiments, the system can use a wireless transmitter in thecell phone that communicates with a central processing unit (CPU)located within an electronic device, such as a computer or the voicecontrolled assistant itself. In this case the CPU of the voicecontrolled assistant controls encryption and decryption of the data onthe hard disk drive. When the wearable transmitter in the cell phone isin range of the receiver in the CPU, the encrypted data is decrypted andstored unencrypted onto the hard disk drive. When the user and wearabletransmitter leave the location, the CPU encrypts the unencrypted dataand saves the encrypted file, and then deletes the unencrypted file.

In another embodiment, the system can restrict the voice controlledassistant power until a portable wireless transmitter is within range.By restricting power to the voice controlled assistant or componentssuch as a disk drive, operation of the voice controlled assistant isdisabled until the transmitter is in range of the device as the deviceis normally in a powered down state.

The user can be communicating with the remote entities via the voicecontrolled assistant. The assistant 104 outputs an audible questions,“What do you want to do?” This output may represent a question from afar end talker, or from a cloud service (e.g., an entertainmentservice). The user is shown replying to the question by stating, “Turnoff all alarms and open all doors” Such command disables all securityfor the home and criminals can then freely take control of the home. Thecamera and speech system are then used to authenticate the user beforedisabling the security system, for example. The smart phone can be usedas another authentication or confirmation before executing a supervisorlevel command to turn off security, for example.

Accordingly, the assistant may be implemented as an aestheticallyappealing device with smooth and rounded surfaces, with some aperturesfor passage of sound waves, and merely having a power cord andoptionally a wired interface (e.g., broadband, USB, etc.). In theillustrated implementation, the assistant 104 has a housing of anelongated cylindrical shape. Apertures or slots are formed in a base endto allow emission of sound waves. A more detailed discussion of oneparticular structure is provided below with reference to FIG. 3 . Onceplugged in, the device may automatically self-configure, or with slightaid of the user, and be ready to use. As a result, the assistant 104 maybe generally produced at a low cost. In other implementations, other I/Ocomponents may be added to this basic model, such as specialty buttons,a keypad, display, and the like.

In one embodiment, the voice controlled assistant has a gateway onboardor alternatively as an add-on device coupled to the housing of theassistant. The gateway's functions are defined at the application layer.This gives the gateway greater responsibilities and capabilities than arouter, which is simply concerned with addressing packets of data. Thegateway can examine the contents of those data packets and block orforward them according to predefined rules set by the networkadministrator. The gateway also translates between disparate protocols.The gateway only has to deal with two networks—the private network andthe external network linking the private network to external systems.The gateway must operate according to the standards of both networks.Because of this dual role, the gateway can act as a translator betweenthe two networks if they operate on different networking standards.Exemplary protocols that are translated include Zigbee, Z-Wave,Bluetooth, Wifi, UWB, among others. The gateway alternatively can beimplemented as a proxy server. There are two types of proxy server:forward and reverse. The reverse proxy is a gateway that acceptsincoming connections on behalf of network equipment serving externalclients. It receives requests on the server's behalf and communicateswith the server by a separate channel, retrieving the requestedinformation. A forward proxy server can act as a firewall, blocking allincoming connections or monitoring and filtering the responses toconnections originating from within the organization. In both instances,the gateway accepts traffic from one side—either the private LAN or theexternal network—and fulfills the request itself, without allowing thetwo sides to connect directly.

In certain embodiments, as part of determining whether the voice querycan be acted upon (i.e., processed), the computing device can determinea threshold level of user authentication that is required based on thequery's security level (e.g., is voice-based authentication sufficient,or are additional forms of authentication required, such as face, PIN,etc.). The computing device can then prompt the user to authenticatehimself/herself using the additional authentication method(s) as neededin order to proceed with query processing. Further, in some embodiments,the step of identifying/authenticating the user can be performed inparallel with the step of recognizing the voice query in order to ensurelow latency operation. These and other aspects of the present disclosureare described in additional detail in the sections that follow.

The invention may be implemented in hardware, firmware or software, or acombination of the three. Preferably the invention is implemented in acomputer program executed on a programmable computer having a processor,a data storage system, volatile and non-volatile memory and/or storageelements, at least one input device and at least one output device.

1. A device including:

a housing having a base to support the housing on a surface and a distaltop end;

at least one microphone mounted proximal to the top end to receiveaudio;

a processor mounted within the housing to process a signalrepresentation of the audio to recognize speech and reduce acousticechoes detected in the signal representation;

at least one speaker mounted in the housing and oriented to output soundin a downward direction toward the base and away from the top end; and

a gateway coupled to the processor to communicate with home appliances,the gateway communicating with a plurality of home area network (HAN)protocols including Zigbee (IEEE 802.15.4), Bluetooth (IEEE 802.15.1)and Z-Wave, the gateway translating HAN data from one protocol toanother;

a first home appliance having a first security level;

a second home appliance having a second security level.

2. The device, further including a speech recognition module executableby the processor to recognize the speech in the signal representation.

3. The device, further including an acoustic echo cancellation moduleexecutable by the processor to reduce the acoustic echoes detected inthe signal representation.

4. The device, including a mobile device with a user key incommunication with the processor, wherein the processor authenticatesthe user key by detecting signals from mobile device.

5. The device, wherein the device communicates with a remote server toanalyze the verbal command.

6. A method including: receiving audio, via one or more microphones, ina device having the one or more microphones positioned at a top end;processing a signal representation of the audio to one or more of (1)recognize speech in the signal representation or (2) cancel acousticechoes detected in the signal representation; and outputting sound viaone or more speakers arranged in a base end of the device, wherein thesound is output from the one or more speakers in a downward directiontoward the base end directionally opposite to the one or moremicrophones at the top end.

7. The method, wherein recognizing the speech comprises processing thesignal representation to parse the speech.

8. The method, further including processing the signal representation toreduce double talk detected in the signal representation, in conjunctionwith cancelling the acoustic echoes detected in the signalrepresentation.

9. The method, wherein processing the signal representation comprisessubstantially cancelling acoustic echoes detected in the signalrepresentation, and then parsing the speech in the signalrepresentation.

10. A device including: a housing having a base to support the housingon a surface and a distal end; at least one microphone mounted at thedistal end of the housing to receive audio; a processor mounted withinthe housing to process a signal representation of the audio to one ormore of (1) recognize speech in the signal representation or (2)substantially cancel acoustic echoes detected in the signalrepresentation; and at least one speaker arranged inside the housing andoriented to output sound in a direction away from the at least onemicrophone.

11. The device, further including a speech recognition module executableby the processor to recognize the speech in the signal representation.

12. The device, wherein the speech comprises specific commands.

13. The device, further including an acoustic echo cancellation moduleexecutable by the processor to substantially cancel the acoustic echoesdetected in the signal representation.

14. The device, wherein the housing has one or more openings near thebase to pass sound waves from the at least one speaker.

15. The device, wherein the at least one speaker comprises a pluralityof speakers that are coaxially aligned.

16. The device, wherein the at least one speaker and the at least onemicrophone are coaxially aligned.

17. The device, further including a sound distribution cone arrangedinside of the housing to distribute the sound emitted from the at leastone speaker.

18. The device wherein the at least one speaker and the sounddistribution cone are coaxially aligned.

19. The device, wherein the sound distribution cone directs the sound atleast partially in a radial outward direction.

20. The device, wherein the sound distribution cone directs the soundoutward from the housing proximal to the base.

1. An appliance, including:

-   -   a body; a processor in the body and coupled to a wireless        transceiver; a camera coupled to the body; and an accelerometer        to detect acceleration;    -   a surface with a sensor to sense characteristics of food or        drink;    -   a motor to affect the food or drink;    -   a wireless transceiver connected to the Internet; and    -   a processor coupled to the wireless transceiver and the sensor,        the processor running a voice recognizer to answer a verbal        query relating to the appliance and the food or drink.

2. The container, including a camera coupled to the processor and one ormore volume indicia on the container to indicate remaining content.

3. The container, including mildew detection code to determine thepresence of mildew in the container.

4. The container, including rechargeable battery or super-capacitorcoupled to a wireless power receiver in the container to receive powerfrom a layer or sheet of wireless power transmitters.

5. The container, wherein the wireless power receiver comprises an NFCreceiver and wherein the layer or sheet of wireless power transmittersreceive container status data.

6. The container, including code to report status including content ageor quality.

7. The container, including code to suggest best ways to enjoy thecontent.

8. The container, including an electronic nose to detect an odorindicating content quality.

9. The container, including code to contact a consumer, distributor,retailer, or manufacturer with a status of the content.

10. The container, including code to exchange information between theconsumer and a distributor, retailer or manufacturer of the content.

11. The container, including a sensor to detect container opening.

12. The container, including an anti-fraud lid and registration code todetect product tampering or product fraud.

13. The container, including a MEMS sensor to detect liquid quality,wherein the MEMS sensor detect liquid conductance or resistance andcompares with a predetermined quality rating.

14. The container, including code to sequence usage of the container ina first-in-first-out (FIFO) order to reduce spoilage.

15. The container, including code to determine if the content is below apredetermined threshold and if so ordering additional supply.

16. The container, including code to determine expiration date andsignaling a user to use prior to the expiration date.

17. A system, including:

an appliance;

a digitally responsive container to store content therein, the containerreceived by the appliance, the container, including:

-   -   a volume sensor for the content;    -   a wireless transceiver; and    -   a processor coupled to the wireless transceiver and the volume        sensor, the processor responding to a query on remaining content        and quality of the content; and

a smart phone, watch, tablet, or mobile computer wirelessly coupled toone or more of the appliances to control settings on the appliance.

18. The system, including a camera coupled to the processor and one ormore volume indicia on the container to indicate remaining content.

19. The system, wherein the container comprises mildew detection code todetermine the presence of mildew in the container.

20. The system, wherein the container comprises rechargeable battery orsuper-capacitor coupled to a wireless power receiver in the container toreceive power from a layer or sheet of wireless power transmitters.

The IOT appliances can communicate using blockchain as described inco-pending application Ser. Nos. 15/594,311 and 15/594,214 both filed onMay 12, 2017, the contents of which are incorporated by reference. TheIoT machines can negotiate contracts on their own (without human) andexchange items of value by presenting an open transaction on theassociated funds in their respective wallets. Blockchain token ownershipis immediately transferred to a new owner after authentication andverification, which are based on network ledgers within a peer-to-peernetwork, guaranteeing nearly instantaneous execution and settlement. Asimilar process is used to provide secure communications between IoTdevices, which is useful for edge IoT devices. The industrial world isadding billions of new IoT devices and collectively these devicesgenerate many petabytes of data each day. Sending all of this data tothe cloud is not only very cost prohibitive but it also creates agreater security risk. Operating at the edge ensures much fasterresponse times, reduced risks, and lower overall costs. Maintainingclose proximity to the edge devices rather than sending all data to adistant centralized cloud, minimizes latency allowing for maximumperformance, faster response times, and more effective maintenance andoperational strategies. In addition to being highly secure, the systemalso significantly reduces overall bandwidth requirements and the costof managing widely distributed networks.

In some embodiments, the described technology provides a peer-to-peercryptographic currency trading method for initiating a market exchangeof one or more Blockchain tokens in a virtual wallet for purchasing anasset (e.g., a security) at a purchase price. The system can determine,via a two-phase commit, whether the virtual wallet has a sufficientquantity of Blockchain tokens to purchase virtual assets (such aselectricity only from renewable solar/wind/ . . . sources, weather dataor location data) and physical asset (such as gasoline for automatedvehicles) at the purchase price. In various embodiments, in response toverifying via the two-phase commit that the virtual wallet has asufficient quantity of Blockchain tokens, the IoT machine purchases (orinitiates a process in furtherance of purchasing) the asset with atleast one of the Blockchain tokens. In one or more embodiments, if thedescribed technology determines that the virtual wallet has insufficientBlockchain tokens for purchasing the asset, the purchase is terminatedwithout exchanging Blockchain tokens.

The present system provides smart contract management with modules thatautomates the entire lifecycle of a legally enforceable smart contractby providing tools to author the contract so that it is bothjudge/arbitrator/lawyer readable and machine readable, and ensuring thatall contractual obligations are met by integrating with appropriateexecution systems, including traditional court system, arbitrationsystem, or on-line enforcement system. Different from theblockchain/bitcoin contract system where payment is made in advance andreleased when the conditions are electronically determined to besatisfied, this embodiment creates smart contracts that are verifiable,trustworthy, yet does not require advance payments that restrict theapplicability of smart contracts.

In addition to Ethereum, other blockchain or globally shared,transactional database can be used. To change something in the database,the system creates a transaction which has to be accepted by all others.One embodiment runs on an Ethereum Virtual Machine or EVM as the runtimeenvironment for smart contracts in Ethereum. It is not only sandboxedbut actually completely isolated, which means that code running insidethe EVM has no access to network, filesystem or other processes. Smartcontracts have limited access to other smart contracts. There are twokinds of accounts in Ethereum which share the same address space:External accounts that are controlled by public-private key pairs (i.e.humans) and contract accounts which are controlled by the code storedtogether with the account. The address of an external account isdetermined from the public key while the address of a contract isdetermined at the time the contract is created (it is derived from thecreator address and the number of transactions sent from that address,the so-called “nonce”). Every account has a persistent key-value storemapping 256-bit words to 256-bit words called storage. Furthermore,every account has a balance in Ether (such as in “Wei”) which can bemodified by sending transactions that include Ether.

A transaction is a message that is sent from one account to anotheraccount (which might be the same or the special zero-account, seebelow). It can include binary data (its payload) and Ether. If thetarget account contains code, that code is executed and the payload isprovided as input data. If the target account is the zero-account (theaccount with the address 0), the transaction creates a new contract. Asalready mentioned, the address of that contract is not the zero addressbut an address derived from the sender and its number of transactionssent (the “nonce”). The payload of such a contract creation transactionis taken to be EVM bytecode and executed. The output of this executionis permanently stored as the code of the contract. This means that inorder to create a contract, you do not send the actual code of thecontract, but in fact code that returns that code. Upon creation, eachtransaction is charged with a certain amount of gas, whose purpose is tolimit the amount of work that is needed to execute the transaction andto pay for this execution. While the EVM executes the transaction, thegas is gradually depleted according to specific rules. Each account hasa persistent memory area which is called storage. Storage is a key-valuestore that maps 256-bit words to 256-bit words. It is not possible toenumerate storage from within a contract and it is comparatively costlyto read and even more so, to modify storage. A contract can neither readnor write to any storage apart from its own. The second memory area iscalled memory, of which a contract obtains a freshly cleared instancefor each message call. Memory is linear and can be addressed at bytelevel, but reads are limited to a width of 256 bits, while writes can beeither 8 bits or 256 bits wide. Memory is expanded by a word (256-bit),when accessing (either reading or writing) a previously untouched memoryword (ie. any offset within a word). At the time of expansion, the costin gas must be paid. The EVM is not a register machine but a stackmachine, so all computations are performed on an area called the stack.It has a maximum size of 1024 elements and contains words of 256 bits.Access to the stack is limited to the top end in the following way: Itis possible to copy one of the topmost 16 elements to the top of thestack or swap the topmost element with one of the 16 elements below it.All other operations take the topmost two (or one, or more, depending onthe operation) elements from the stack and push the result onto thestack. Of course it is possible to move stack elements to storage ormemory, but it is not possible to just access arbitrary elements deeperin the stack without first removing the top of the stack.

The instruction set of the EVM is kept minimal in order to avoidincorrect implementations which can cause consensus problems. Allinstructions operate on the basic data type, 256-bit words. The usualarithmetic, bit, logical and comparison operations are present.Conditional and unconditional jumps are possible. Furthermore, contractscan access relevant properties of the current block like its number andtimestamp.

Contracts can call other contracts or send Ether to non-contractaccounts by the means of message calls. Message calls are similar totransactions, in that they have a source, a target, data payload, Ether,gas and return data. In fact, every transaction consists of a top-levelmessage call which in turn can create further message calls.

A contract can decide how much of its remaining gas should be sent withthe inner message call and how much it wants to retain. If an out-of-gasexception happens in the inner call (or any other exception), this willbe signalled by an error value put onto the stack. In this case, onlythe gas sent together with the call is used up. In Solidity, the callingcontract causes a manual exception by default in such situations, sothat exceptions “bubble up” the call stack.

As already said, the called contract (which can be the same as thecaller) will receive a freshly cleared instance of memory and has accessto the call payload—which will be provided in a separate area called thecalldata. After it finished execution, it can return data which will bestored at a location in the caller's memory preallocated by the caller.

With a message call, named delegatecall which is identical to a messagecall apart from the fact that the code at the target address is executedin the context of the calling contract and msg.sender and msg.value donot change their values, a contract can dynamically load code from adifferent address at runtime. Storage, current address and balance stillrefer to the calling contract, only the code is taken from the calledaddress.

Contracts can create other contracts using a special opcode (i.e. theydo not simply call the zero address). The only difference between thesecreate calls and normal message calls is that the payload data isexecuted and the result stored as code and the caller/creator receivesthe address of the new contract on the stack. More information onSolidity is at Introduction to Smart Contracts athttp://solidity.readthedocs.io/en/develop/introduction-to-smart-contracts.html,the content of which is incorporated by reference.

The system enables the physical goods and materials to be identified andlinked with their digital representation on the blockchain (e.g., serialnumbers, bar codes, digital tags like RFID and NFC, genetic tags) iscrucial in uniquely identifying a physical good with its digitalcounterpart. At Provenance we are exploring many new and existingtechnologies; an overview of recent technologies can be found here.Identities are recorded in production and manufacturing programs, andfor simplicity and easy adoption we expect them to take the form ofexisting barcodes and serial numbers which are linked to blockchainidentifiers using a secure hash.

User-facing applications facilitate access to the blockchain. The finalowner of the product has access to secure information about theproduct's supply chain, without having access to identification details.The final owner of the product has access to secure information aboutthe product's supply chain, without having access to identificationdetails.

By design, every transaction along a supply chain on the blockchain isfully auditable. By inspecting the blockchain, smartphone applicationscan aggregate and display information to customers in a real-timemanner; furthermore, due to the strong integrity properties of theblockchain, this information can be genuinely trusted. A user interfacesheds light on the digital journey of a product can empower betterpurchases by giving users a true choice that they can exercise. Thereare substantial broad effects of bringing near-frictionless transparencyto consumer purchase decisions and product identity; clearly there islikely to be an additional “virtuous” component in purchase decisions,especially among mid-level purchases where a marginal increase of 20% tothe price does not affect the willingness to buy. Additional levels ofguarantee over genuine articles is a high-value use case. While aninitial introduction of this technology may be in the form of a discreteand removable label, easily verified through a smartphone-readableQR-code, a more progressive possibility would be a conspicuoushologramatic or RFID tag, embedded in the brand label, allowing theowner to prove the authenticity of the product at any time by accessingthe data on the blockchain through the tag.

In the system, everyone has a profile accessible with a private key.Profiles can be public or private depending on use case and permissions.Some are rich with information, whilst others simply contain ananonymous ID. The system supports the registration of named participants(i.e. certifiers, auditors, producers, and manufacturers). Suchparticipants may request registration of their digital identity whichlinks their real-world identity with their blockchain-based digitalidentity, thus allowing them to interact with the blockchain using theirreal-world identity. Upon request, the registration authority verifiestheir identity and records the result in the blockchain, available forall to inspect.

These programs represent the implementation of schemas for properrecognition of a standard (e.g. no animal testing, biodynamic, fairlabor). Through these programs, standards organizations provide for thecreation of compliant production or manufacturing programs (see below),allowing instances or batches of goods and materials to be added to orprocessed on the blockchain. Such producers or manufacturers may requireinspection by a certifier or auditor of their facilities and processesto be able to obtain and operate a certified program. Successfulverification results in the deployment of a production or manufacturingprogram that is both registered with the certification program andauthenticated by an auditor, and allows a producer to create thedigitally tradeable equivalent of a good (i.e., a token that shadows thereal-world material or product).

The physical goods and materials are identified and linked with theirdigital representation on the blockchain using a label (e.g., serialnumbers, bar codes, digital tags like RFID and NFC, genetic tags) thatuniquely identifies a physical good with its digital counterpart.Identities are recorded in production and manufacturing programs, andfor simplicity and easy adoption the system can use electronic tags orbarcodes and serial numbers which are linked to blockchain identifiersusing a secure hash. During manufacturing, each item is associated witha tag. The tag can be a discrete and removable label, easily readthrough a smartphone-readable QR-code, a hologram or RFID tag, embeddedin the brand label, allowing the owner to prove the authenticity of theproduct at any time by accessing the data on the blockchain through thetag. While a tag such as a bar code cannot store information, it cansave information to a remote server that associates that tag withvarious blockchains. Alternatively, active memory can be formed usingroll-to-roll electronic printing onto a tag and the app can store theposition information and additional information to a circuit such asflexible circuit, a printed circuit, or an electronic tag with memory,and the tag can be associated with a product 3 as it moves throughproduction and shipping processes. In one embodiment, the circuit can be“printed memory” that can collect and store information about theauthenticity and condition of products. One embodiment uses a PrintedMemory containing up to 36 bits of rewritable memory which can store upto 68 billion points of data. The labels are used to determine if aproduct is genuine and to track how it's been handled duringdistribution. Another embodiment uses Printed Memory with CryptographicSecurity that includes a unique, encrypted printed code (such as a QRbar code) to the memory. It can only be read by authorized personnelusing a reader which interfaces with a secure smartphone application.This combination of printed memory with an encrypted printed code,creates a secure anti-counterfeit solutions. This makes it possible toensure the integrity of a product from the time it leaves the factory tothe time it gets into the hands of a customer with a cost efficient,highly secure method of authenticating and verifying information about aproduct as it moves through various distribution channels or as it isused.

Augmented Reality/Virtual Reality Sports Gaming

FIG. 15 shows an exemplary 360 degree camera on a helmet, for example,for augmenting reality of sport games. Using augmented reality, variousways may exist for a user to “participate” in a live event. Generally,augmented reality refers to a presentation of a real world environmentaugmented with computer-generated data (such as sound, video, graphicsor other data). In some embodiments, augmented reality, implemented inconjunction with a live event, may allow a user to control a virtualobject that appears to compete or otherwise interact with theparticipants of the live event. For example, an end user device, such asa mobile phone, tablet computer, laptop computer, or gaming console maybe used to present a live video feed of an event to a user. This livevideo feed may be video of an event that is occurring in real-time,meaning the live event is substantially concurrently with thepresentation to the user (for example, buffering, processing, andtransmission of the video feed may result in a delay anywhere from lessthan a second to several minutes). The presentation of the live eventmay be augmented to contain one or more virtual objects that can be atleast partially controlled by the user. For instance, if the live eventis a stock car race, the user may be able to drive a virtual cardisplayed on the end user device to simulate driving in the live eventamong the actual racers. As such, the user may be able to virtually“compete” against the other drivers in the race. The virtual object, inthis example a car, may be of a similar size and shape to the real carsof the video feed. The user may be able to control the virtual car torace against the real cars present in the video feed. The real carsappearing in the video feed may affect the virtual object. For example,the virtual object may not be allowed to virtually move through a realcar on the augmented display, rather the user may need to drive thevirtual object around the real cars. Besides racing, similar principlesmay be applied to other forms of live events; for example, track andfield events (e.g., discus, running events, the hammer toss, polevaulting), triathlons, motorbike events, monster truck racing, or anyother form of event that a user could virtually participate in againstthe actual participants in the live event. In some embodiments, a usermay be able to virtually replay and participate in past portions of alive event. A user that is observing a live event may desire to attemptto retry an occurrence that happened during the live event. Whileviewing the live event, the user may be presented with or permitted toselect an occurrence that happened in the course of the live event andreplay it such that the user's input affects the outcome of at leastthat portion of the virtualized live event. Using a baseball game as anexample, with runners on first and third, two outs, and the count beingtwo balls and two strikes, the pitcher may throw a splitter,successfully striking out the batter with a pitch in the dirt. Theinning may end and the game may continue. The user may desire to replaythis unsuccessful at-bat with himself controlling the batter during thecommercial break. As such, via an end user device, the user may be ableto indicate the portion of the game he wishes to replay (e.g., the lastat-bat). Game facts from the live event may be used to virtuallyrecreate this at-bat for the user. For instance, the virtual game loadedby the user may use game facts leading up to the at-bat the user hasselected. For instance, the opposing team, the stadium, the score, thetime of day, the batter, the pitcher, and the sequence of pitches thrownby the pitcher may be used to provide the user with a virtual replay ofat least that portion of the baseball game that the user can affect viainput (e.g., swinging and aiming the virtual bat). In replaying theselected portion of the live event, the entire event may be virtualized.As such, referring to the baseball example, the pitcher, stadium, field,fielders, batter, and ball may all be replaced by virtual objects, withone (or more) of the virtual objects, such as the batter, beingcontrolled by the user. As such, this may resemble a video gameinstantiated with data from the live event. In some embodiments, aportion of the live event may involve a playback of a video feed of thelive event with a virtual object that is controlled by the user beingaugmented. Referring again to the example of the baseball game, thepitcher, stadium, fielders, and field may be replayed from the videofeed; the batter and/or ball may be virtualized. As such, the user maycontrol the batter and swing at a virtual ball that has taken the placeof the real ball present in the video feed. Besides baseball, suchreenactment of a portion of a live event may be applied to various formsof sporting events, such as football, soccer, tennis, golf, hockey,basketball, cricket, racing, skiing, gymnastics, and track and fieldevents. Other forms of live events, besides sports, may also bereenacted using such techniques.

FIG. 15A shows a multi-headed camera array 423 that may be at least partof a modular camera system, with each camera forming a module of themodular camera system. The camera array has a flexible structure so thatit is easy to remove a particular camera module from the camera arrayand to add new camera modules to the camera array. The camera modules inthe camera array may be configured in different geometries. For example,the camera array includes multiple camera modules arranged in a line, acylinder, a sphere, or another geometry. Each camera module may beconfigured to point to a different direction so that the camera arraymay capture an object or a scene from multiple directions at the sametime.

The camera system described herein may additionally include a set ofalgorithms for processing the video data captured by the camera array.The set of algorithms are stored on a non-transitory memory forconverting the input across multiple camera modules into a single streamof 3D video (e.g., a single compressed stream of 3D video data). The setof algorithms may be implemented in one or more “modules”. For example,the set of algorithms includes color correction algorithms for smoothingand correcting colors in the video data. In another example, the set ofalgorithms may be implemented in software that stitches the video datafrom multiple cameras into two large-format, panoramic video streams forleft and right eye viewing, and encodes and compresses the video using astandard MPEG format or other suitable encoding/compression format.

The camera array 423 may be constructed using various configurations.For example, the camera modules may be configured in differentgeometries (e.g., a sphere, a line, a cylinder, a cone, a cube, etc.)with the corresponding lenses 113 facing in different directions. Forexample, the camera modules are positioned within the camera array 423in a honeycomb pattern where each of the compartments form an aperturewhere a camera module may be inserted. In another example, the cameraarray 423 includes multiple lenses along a horizontal axis and a smallernumber of lenses on a vertical axis.

In some embodiments, the camera modules in the camera array 423 areoriented around a sphere in different directions with sufficientdiameter and field-of-view to capture enough view disparity to renderstereoscopic images.

The camera array 423 has a flexible structure so that a particularcamera module may be removed from the camera array 423 easily. In someembodiments, the camera modules are rotationally symmetrical such that acamera module may be inserted into the housing, removed, rotated 90degrees, and reinserted into the housing. In this example, the sides ofthe housing may be equidistant, such as a camera module with fourequidistant sides. This allows for a landscape orientation or a portraitorientation of the image frames without changing the base. In someembodiments, the lenses and the camera modules are interchangeable. Newcamera modules may also be added to the camera array 423. In someembodiments, the camera modules in the camera array 423 are positionedto have a sufficient field-of-view overlap so that all objects can beseen by more than one view point. In some embodiments, having the cameraarray 423 configured so that an object may be viewed by more than onecamera may be beneficial for correcting exposure or color deficienciesin the images captured by the camera array 423. Other benefits includedisparity/depth calculations, stereoscopic reconstruction, and thepotential to perform multi-camera high-dynamic range (HDR) imaging usingan alternating mosaic pattern of under- and over-exposure across thecamera array.

In some embodiments, the camera array 423 may also include a microphonearray for capturing sound from all directions. For example, themicrophone array may include a Core Sound Tetramic soundfieldtetrahedral microphone array following the principles of ambisonics,enabling reconstruction of sound from any arbitrary direction. Inanother example, the microphone array includes the Eigenmike, whichadvantageously includes a greater number of microphones and, as aresult, can perform higher-order (i.e. more spatially accurate)ambisonics. The microphone may be mounted to the top of the camera array423, be positioned between camera modules, or be positioned within thebody of the camera array 423. The result can then be rendered as animmersive video and a user can view the video with computer annotationsthereon for augmented reality purposes. In one implementation, the eventmay be a live event, for example, but is not limited to, a footballmatch, a cricket match, a basketball match, a theatre, a concert, andthe like. In one embodiment, the augmented reality content may include,but is not restricted to, live content associated with an event,recorded content associated with an event, a curated content, anadvertising content, or a combination thereof. In another embodiment,the augmented reality content may include, but is not restricted to,information related to a service available at an event, a venue of anevent, a status of a service, or a combination thereof. The system 100may also provide the augmented reality content associated with, but isnot restricted to, a venue of an event, duration of an event, a locationof an event, or a combination thereof, in another implementation.

One embodiment allows combined augmented reality and virtual reality onthe display. The method may include selectively allowing a transmissionof light from a local environment of the user based on a visualizationmode of the display object. The visualization mode may be one of anaugmented reality mode, a virtual reality mode, and a combination ofaugmented and virtual reality modes.

In another embodiment, sensors may be placed to track eye movement aswell as hand gestures and verbal commands. The method may furthercomprise capturing a field-of-view image of each of the user's eyes. Thecaptured field of view image may be used to estimate a head pose of theuser. The captured field-of-view image may be used to convert at leastone physical object to a physically rendered virtual object, and todisplay the physically rendered virtual object to the user. In anotherembodiment, sensors may be placed to track eye movement as well as handgestures and verbal commands. Then, a method comprises tracking amovement of a user's eyes, estimating a depth of focus of the user'seyes based on the tracked eye movement, modifying a light beamassociated with a display object based on the estimated depth of focussuch that the display object appears in focus, and projecting themodified light beam into the user's eyes. The diameter of the projectedlight beam projected to the user's eyes may be less than 0.7 mm.

For the athlete/participant who wish to enhance their gaming viaaugmented or virtual reality, features may include the following:

1. A method for using augmented reality, the method including:receiving, by a computerized device, a data stream with a 360 degreeview of a live event on each participant, wherein the data streamcomprises live video augmented with positions of team mates and opposingplayers and recommends a play routine based on live field condition andpositions of other players, wherein the user can select a point of viewfrom a selected participant.

2. The method for using augmented reality, wherein the user plays in avirtual reality version of the live event.

3. The method for using augmented reality, wherein the live event is asporting event.

4. The method, wherein the live event comprises: soccer, football,basketball, tennis, boxing, car racing, golf, ice hockey, badminton,volleyball, cycling, swimming, snooker, martial arts, rugby, motorbike,hockey, table tennis, horse racing, gymnastics, handball, figureskating, wrestling, skiing, diving, skating, archery, sailing,wrestling, fencing, equestrian, rowing, surfing, Beach Volleyball,Pool/Billiards, Lacrosse, Windsurfing, Polo, Tenpin Bowling,Racquetball, Competitive Climbing, Mountain Biking.

FIG. 15 shows an exemplary recommender to aid an athlete in improvingthe game. For example, the process can recommend a strategy in light ofthe opponent's historical performance. In tennis, a player's historicalweakness can be ascertained and a recommendation can be made to optimizesuccess. In a football example, a fourth down module 400 may include aFootball recommender, a Field Goal algorithm, and a Punt algorithm. TheFootball recommender determines the probability of each potential playoutcome associated with the Go For It coaching decision. The Field Goalalgorithm determines the probability of each potential play outcomeassociated with the Field Goal coaching decision. The Punt algorithm1102 determines the probability of each potential play outcomeassociated with the Punt coaching decision. As shown in FIG. 15B, theFootball recommender 402 determines the probability of each potentialplay outcome associated with the Go For It coaching decision on a fourthdown play. The Football recommender 402 receives an expected points (EP)input from the expected points module 300 at block 404, a yards to gain(YTG) for first down input at block 406, and a first down marker (FDM)yard line input at block 408. Preliminary Conversion Rate: At block 410,the Football recommender 402 uses the team's EP value from block 404 andthe YTG distance from block 406 to determine a preliminary first downconversion rate based on historical conversion data. Historical firstdown conversion data is shown in the form of a chart in FIG. 5 , whereYTG distances are presented on the x-axis and average first downconversion rates are presented on the y-axis. This historical data showsthat the likelihood of a first down conversion decreases as the YTGdistance increases. Individual lines or equations may be presented toaccount for various EP values. For simplicity, FIG. 5 shows three linesto account for scenarios in which the offense and defense are an equalmatch with the same EP values (NEU), the offense has the advantage (OFFAD), and the defense has the advantage (DEF AD). The historical datapresented in FIG. 5 shows that stronger offenses will convert firstdowns versus weaker defenses (OFF AD) more often than weaker offenseswill convert first downs versus stronger defenses (DEF AD). Similarlines may be provided for specific EP values (e.g., 7-66 points). Bydetermining the first down conversion rate at each YTG distance for eachoffensive match-up, the Football recommender 402 is able to predict thelikelihood of a first down conversion with great precision.

Inside an opponent's 20-yard line (i.e., in the Red Zone), it becomesmore difficult to convert for a first down as the space on the fieldfrom which to work becomes more limited. As the FDM gets closer to theend zone and the YTG distance increases, the challenge of converting afirst down gets progressively more difficult versus similar scenariosoutside of the Red Zone. To account for the challenge of converting afirst down in the Red Zone, the Football recommender 402 may multiplythe preliminary conversion rate by a field position multiplier at block412 based on the YTG distance from block 406 and the FDM yard line fromblock 408 (where 100 represents the opponent's goal line. As an example,take a team that normally has a 50% fourth down conversion rate with 2YTG. If the team faces a fourth down play with 2 YTG outside of the RedZone, the conversion rate may remain at 50%. However, if the team facesa fourth down play with 2 YTG in the Red Zone, such as from theopponent's 2-yard line when the FDM is on the opponent's goal line(FDM=100), the normal 50% conversion rate may be multiplied by thecorresponding field position multiplier of 85.5% to arrive at a loweradjusted conversion rate of 42.7%. The process may adjust team's firstdown conversion rate at block 412 based on particular strengths of histeam. In one embodiment, the Football recommender 402 multiplies theconversion rate by one or more additional multipliers, such as a YTGmultiplier, which may be specified by the coach. As an example, a teamthat thrives on running the football might find that it convertsshort-yardage situations particularly well, because its offense isdesigned to consistently grind out short gains. However, the same teammay have particular difficulty in converting longer-yardage situationsbecause the offense isn't conducive to big plays. In this example, theYTG multiplier may be greater than 100% below 5 YTG to increase theconversion rate in short-yardage situations and less than 100% above 5YTG to decrease the conversion rate in long-yardage situations.Conversely, a team with an explosive offense may be particularlyeffective in converting long yardages but may not have the personnel toget short yardage. In this example, the YTG multiplier may be less than100% below 5 YTG to decrease the conversion rate in short-yardagesituations and greater than 100% above 5 YTG to increase the conversionrate in long-yardage situations. The Indianapolis Colts were a greatexample of this during much of the Peyton Manning era. They were verydangerous in long-yardage situations due to the quality of their passinggame, but due to a poor running game, they often failed to convert inshort-yardage scenarios. The Football recommender 402 may calculate theprobability of a turnover and defensive touchdown as a function of theEP value from block 404 and the FDM yard line from block 408. Thisprobability may be as low as about 0.1% and as high as about 0.5%. Atblock 414, the Football recommender 402 assigns probabilities to eachpotential conversion outcome. The Football recommender 402 may determinenot only the likelihood of a first down conversion at block 412, butalso how likely the team is to score points if the conversion issuccessful at block 416. After a successful conversion, the team couldget just enough yards to get the first down and still not score anypoints on the drive, or it could score a touchdown on the very same playor a subsequent play of the same drive. Therefore, the Footballrecommender 402 may take into account the potential upside of the driveshould the fourth down play be successful at any field position. Atblock 416, the Football recommender 402 uses the team's EP value fromblock 404 and the FDM yard line from block 408 to determine the pointsscored given conversion based on historical scoring data. Historicalscoring data is shown in the form of a chart in FIG. 6 , where FDM yardlines are presented on the x-axis (with 0 representing the team's owngoal line and 100 representing the opponent's goal line) and averagepoints scored given conversion are presented on the y-axis. Thishistorical data shows that the likelihood of scoring points increases asthe FDM approaches the opponent's goal line. Individual lines orequations may be presented to account for various EP values. Forsimplicity, FIG. 6 shows three lines to account for scenarios in whichthe offense and defense are an equal match with the same EP values(NEU), the offense has the advantage (OFF AD), and the defense has theadvantage (DEF AD). The historical data presented in FIG. 6 shows thatstronger offenses will score more points versus weaker defenses (OFF AD)than weaker offenses will score versus stronger defenses (DEF AD).Similar lines may be provided for specific EP values (e.g., 7-66points). In this manner, the augmented reality system can enhance thegame.

For viewers who wish to participate via augmented or virtual reality,features may include the following:

1. A method for using augmented reality, the method including:receiving, by a computerized device, a data stream with a 360 degreeview of a live event on each participant, wherein the data streamcomprises live video, wherein: the live video comprises a live object;receiving, by the computerized device, input from a user, wherein theinput from the user affects behavior of a virtual object; andpresenting, by the computerized device, the live event augmented by thevirtual object, wherein a behavior of the live object of the live eventaffects the behavior of the virtual object and each participant, whereinthe user can select a point of view from a selected participant.

2. The method for using augmented reality of claim 1, wherein: thevirtual object is presented such that the virtual object appears tocompete with the live object.

3. The method for using augmented reality of claim 1, wherein the liveevent is a sporting event.

4. The method for using augmented reality of claim 1, further including:receiving, by the computerized device, data corresponding to a secondvirtual object from a remote computerized device; and displaying, by thecomputerized device, the live event augmented by the virtual objectfurther augmented with the second virtual object.

5. The method for using augmented reality of claim 4, wherein thebehavior of the second virtual object is affected by a second user.

6. The method for using augmented reality of claim 4, further including:modifying, by the computerized device, behavior of the virtual object inresponse to the second virtual object.

7. A method for using augmented reality, the method including:receiving, by a computerized device, data corresponding to a live event;presenting, by the computerized device, the live event up to a point intime; presenting, by the computerized device, a virtual event at leastpartially based on an event that occurred during the live event earlierthan the point in time; receiving, by the computerized device, inputlinked with the virtual event, wherein the input is received from auser; and presenting, by the computerized device, an outcome of thevirtual event, wherein the outcome is at least partially based on theinput received from the user.

8. The method for using augmented reality of claim 7, wherein: thevirtual event is presented at least starting when the live event isstopped.

9. The method of claim 7, wherein the live event is a sporting event.

10. The method of claim 7, wherein the live event comprises: soccer,football, basketball, tennis, boxing, car racing, golf, ice hockey,badminton, volleyball, cycling, swimming, snooker, martial arts, rugby,motorbike, hockey, table tennis, horse racing, gymnastics, handball,figure skating, wrestling, skiing, diving, skating, archery, sailing,wrestling, fencing, equestrian, rowing, surfing, Beach Volleyball,Pool/Billiards, Lacrosse, Windsurfing, Polo, Tenpin Bowling,Racquetball, Competitive Climbing, Mountain Biking.

11. A method for using virtual reality, the method including: receiving,by a computerized device, a data stream with a 360 degree view of acomputer generated event on each participant, wherein the data streamcomprises live video, wherein: the live video comprises a live object;receiving, by the computerized device, input from a user, wherein theinput from the user affects behavior of a virtual object; andpresenting, by the computerized device, the live event augmented by thevirtual object, wherein a behavior of the live object of the live eventaffects the behavior of the virtual object and each participant.

12. A method for using augmented reality and virtual reality, the methodincluding: receiving, by a computerized device, a data stream with a 360degree view of a live event on each participant, wherein the data streamcomprises live video, wherein: the live video comprises a live object;receiving, by the computerized device, input from a user, wherein theinput from the user affects behavior of a virtual object; andpresenting, by the computerized device, the live event augmented by thevirtual object, wherein a behavior of the live object of the live eventaffects the behavior of the virtual object and each participant, andwherein the virtual reality is rendered by switching the display from anaugmented view to a virtual reality view by fading out the augmentedview on the display to show only the virtual reality view and switchingback when augmented reality view is desired.

Moreover, the viewers can collaboratively read the situation andrecommend a strategy in real-time to improve viewer participation. Inthis manner,

1. A method for participating in a game, the method including:collecting from viewers of a game one or more state change events duringa game; determining whether a series of the collected state changeevents are a known pattern; requesting, when the series of the collectedstate change events is an unknown pattern, viewers of the game toidentify what caused the collected state change events; and judging, bythe viewers, a best reason among the identified causes of the collectedstate change events.2. The method including running a lottery to decide which recommendationis used for the next play in the game.3. The method including: compensating at least one viewer who is judgedto have the best reason among the identified causes of the collectedstate change events.4. The method including: storing as the known pattern, the best reasonamong the identified causes of the collected state change events whenone of the pattern is repeated greater than a threshold number ofrepeats, and the number of the viewers who agree with the correspondingbest reason is greater than a threshold number of users.5. The method including: associating with the stored best reason acorrective action to be taken in response to a future corresponding thecollected state change events.6. The method including: displaying to the other viewers and players,when the stored best reason is known the occurrence of the stored bestreason.7. The method including: transmitting the stored best reason to otherviewers.8. The method wherein the series of the collected state change eventsare at least two state change events that occur within a thresholdperiod of time from each other.Recognition of Exercise Pattern and Tracking of Calorie Consumption

FIG. 16A illustrates the positions of a ski 126′ and skier 128′ during alofting maneuver on the slope 132′. The ski 126′ and skier 128′ speeddown the slope 132′ and launch into the air 136 at position “a,” andlater land at position “b” in accord with the well-known Newtonian lawsof physics. With an airtime sensor, described above, the unit 10calculates and stores the total airtime that the ski 126′ (and hence theskier 128′) experiences between the positions “a” and “b” so that theskier 128′ can access and assess the “air” time information. Airtimesensors such as the sensor 14 may be constructed with known components.Preferably, the sensor 14 incorporates either an accelerometer or amicrophone. Alternatively, the sensor 14 may be constructed as amechanical switch that detects the presence and absence of weight ontothe switch. Other airtime sensors 14 will become apparent in thedescription which follows. The accelerometer sensesvibration—particularly the vibration of a vehicle such as a ski ormountain bike—moving along a surface, e.g., a ski slope or mountain biketrail. This voltage output provides an acceleration spectrum over time;and information about airtime can be ascertained by performingcalculations on that spectrum. Based on the information, the system canreconstruct the movement path, the height, the speed, among others andsuch movement data is used to identify the exercise pattern. Forexample, the skier may be interested in practicing mogul runs, and thesystem can identify foot movement and speed and height information andpresent the information post exercises as feedback. Alternatively, thesystem can make live recommendations to improve performance to theathlete.

FIG. 16B illustrates a sensing unit 10″ mounted onto a mountain bike138. FIG. 16B also shows the mountain bike 138 in various positionsduring movement along a mountain bike race course 140 (for illustrativepurposes, the bike 138 is shown without a rider). At one location “c” onthe race course 140, the bike 138 hits a dirt mound 142 and catapultsinto the air 144. The bike 138 thereafter lands at location “d”. Asabove, with speed and airtime sensors, the unit 10 provides informationto a rider of the bike 138 about the speed attained during the ridearound the race course 140; as well as information about the airtimebetween location “c” and “d”. In this case, the system can recommend acadence to be reached by the rider, strengthen of abdominals, back andarms, for example.

For golf exercise, It is beneficial to require the golfer to swing thegolf club a plurality of times at each swing position to account forvariations in each swing. The swing position at which the golf club isswung can be determined by analysis of the measured accelerationprovided by the accelerometer, e.g., the time at which the accelerationchanges. Data obtained during the training stage may be entered into avirtual table of swing positions and estimated carrying distances for aplurality of different swing positions and a plurality of differentswings. A sample format for such a table is as follows, and includes theaveraged carrying distance for each of four different swing positions.The swing analyzer provides a golfer with an excellent estimation of thecarrying distance of a golf ball for a golf club swing at a specificswing position because it has been trained on actual swings by thegolfer of the same club and conversion of information about these swingsinto estimated carrying distances. The golfer can improve their golfgame since they can better select a club to use to hit a golf club fordifferent situations during a round of golf. Also, the swing pattern isused to identify each club path responsible for the curve of any shotand this information is used to improve the golfer. The direction of theclub path relative to the target, out-to-in (fade pattern) or in-to-out(draw pattern), is what I refer to as a players swing pattern. Playersthat swing from in-to-out will tend to hit draws and players that swingfrom out-to-in will tend to hit fades. Where the ball is struck on theface of the driver (strike point) can drastically alter the effect of aplayers swing pattern on ball flight. Thus, the camera detects where theball is struck, and a computer physics model of ball behavior ispresented to the golfer to improve the score. Shots struck off the heelwill tend to fade more or draw less and shots struck off the toe willtend to draw more or fade less. Thus, camera images of the shots struckof heel or toe can also be used to provide patternrecognition/prediction and for training purposes.

For tennis, examples of motions determined for improvement are detailednext. The system can detect if the continental grip is achieved.Throwing Action pattern is also detected, as the tennis serve is anupwards throwing action that would deliver the ball into the air if itwere a baseball pitch. Ball Toss improvements can be determined when theplayer lines the straight arm up with the net post and release the ballwhen your hand reaches eye level. The system checks the forwarddirection so the player can drive weight (and built up momentum) forwardinto the ball and into the direction of the serve.

The sensors can work with a soccer training module with kinematics ofball control, dribbling, passing, crossing, shooting, heading,volleying, taking throw-ins, penalties, corner kicks and free kicks,tackling, marking, juggling, receiving, shielding, clearing, andgoalkeeping. The sensors can work with a basketball training module withkinematics of crossover dribble, behind back, pull back dribble, lowdribble, basic dribble, between legs dribble, Overhead Pass, Chest Pass,Push Pass, Baseball Pass, Off-the-Dribble Pass, Bounce Pass, Jump Shot,Dunk, Free throw, Layup, Three-Point Shot, Hook Shot.

The sensors can work with a baseball training module with kinematics ofHitting, Bunting, Base Running and Stealing, Sliding, Throwing, FieldingGround Balls, Fielding Fly Balls, Double Plays and Relays, Pitching andCatching, Changing Speeds, Holding Runners, Pitching and PitcherFielding Plays, Catching and Catcher Fielding Plays.

For weight training, the sensor can be in gloves as detailed above, orcan be embedded inside the weight itself, or can be in a smart watch,for example. The user would enter an app indicating that the user isdoing weight exercises and the weight is identified as a dumbbell, acurl bar, and a bar bell. Based on the arm or leg motion, the systemautomatically detects the type of weight exercise being done. In oneembodiment shown in FIG. 15C, with motion patterns captured by glove andsock sensors, the system can automatically detect the followingexemplary exercise:

Upper Body:

Chest: Barbell Bench Presses, Barbell Incline Presses, Dumbbell BenchPresses, Dumbbell Incline Presses, Dumbbell Flyes, Cable Crossovers

Back: Pull-Ups, Wide-Grip Lat Pulldowns, One-Arm Dumbbell Rows, SeatedCable Rows, Back Extensions, Straight Arm Pulldowns

Shoulders: Seated Dumbbell Presses, Front Raises, Lateral Raises,Reverse Flyes, Upright Cable Rows, Upright Barbell Rows

Biceps: Alternate Dumbbell Curls, Barbell Curls, Preacher Curls,Concentration Curls, Cable Curls, Hammer Curls

Triceps: Seated Triceps Presses, Lying Triceps Presses, TricepsKickbacks, Triceps Pushdowns, Cable Extensions, Bench Dips Lower Body

Quadriceps: Barbell Squats, Leg Presses, Leg Extensions

Hamstrings: Dumbbell Lunges, Straight-Leg Deadlifts, Lying Leg Curls

Calves: Seated Calf Raises, Standing Heel Raises

Abs: Floor Crunches, Oblique Floor Crunches, Decline Crunches, DeclineOblique,

Hanging Knee Raises, Reverse Crunches, Cable Crunches, Cable ObliqueCrunches

In one implementation in FIG. 16D, an HMM is used to track weightliftingmotor skills or sport enthusiast movement patterns. Human movementinvolves a periodic motion of the legs. Regular walking involves thecoordination of motion at the hip, knee and ankle, which consist ofcomplex joints. The muscular groups attached at various locations alongthe skeletal structure often have multiple functions. The majority ofenergy expended during walking is for vertical motion of the body. Whena body is in contact with the ground, the downward force due to gravityis reflected back to the body as a reaction to the force. When a personstands still, this ground reaction force is equal to the person's weightmultiplied by gravitational acceleration. Forces can act in otherdirections. For example, when we walk, we also produce friction forceson the ground. When the foot hits the ground at a heel strike, thefriction between the heel and the ground causes a friction force in thehorizontal plane to act backwards against the foot. This force thereforecauses a breaking action on the body and slows it down. Not only dopeople accelerate and brake while walking, they also climb and dive.Since reaction force is mass times acceleration, any such accelerationof the body will be reflected in a reaction when at least one foot is onthe ground. An upwards acceleration will be reflected in an increase inthe vertical load recorded, while a downwards acceleration will bereduce the effective body weight. Zigbee wireless sensors with tri-axialaccelerometers are mounted to the sport enthusiast on different bodylocations for recording, for example the tree structure as shown in FIG.16D. As shown therein, sensors can be placed on the four branches of thelinks connect to the root node (torso) with the connected joint, leftshoulder (LS), right shoulder (RS), left hip (LH), and right hip (RH).Furthermore, the left elbow (LE), right elbow (RE), left knee (LK), andright knee (RK) connect the upper and the lower extremities. Thewireless monitoring devices can also be placed on upper back body nearthe neck, mid back near the waist, and at the front of the right legnear the ankle, among others.

The sequence of human motions can be classified into several groups ofsimilar postures and represented by mathematical models calledmodel-states. A model-state contains the extracted features of bodysignatures and other associated characteristics of body signatures.Moreover, a posture graph is used to depict the inter-relationshipsamong all the model-states, defined as PG(ND, LK), where ND is a finiteset of nodes and LK is a set of directional connections between everytwo nodes. The directional connection links are called posture links.Each node represents one model-state, and each link indicates atransition between two model-states. In the posture graph, each node mayhave posture links pointing to itself or the other nodes.

In the pre-processing phase, the system obtains the human body profileand the body signatures to produce feature vectors. In the modelconstruction phase, the system generate a posture graph, examinefeatures from body signatures to construct the model parameters of HMM,and analyze human body contours to generate the model parameters ofASMs. In the motion analysis phase, the system uses features extractedfrom the body signature sequence and then applies the pre-trained HMM tofind the posture transition path, which can be used to recognize themotion type. Then, a motion characteristic curve generation procedurecomputes the motion parameters and produces the motion characteristiccurves. These motion parameters and curves are stored over time, and ifdifferences for the motion parameters and curves over time is detected,the system then runs the sport enthusiast through additional tests toconfirm the detected motion.

In one exemplary process for determining exercise in the left or righthalf of the body, the process compares historical left shoulder (LS)strength against current LS strength (3200). The process also compareshistorical right shoulder (RS) strength against current RS strength(3202). The process can compare historical left hip (LH) strengthagainst current LH strength (3204). The process can also comparehistorical right hip (RH) strength against current RH strength (3206).If the variance between historical and current strength exceedsthreshold, the process generates warnings (3208). Furthermore, similarcomparisons can be made for sensors attached to the left elbow (LE),right elbow (RE), left knee (LK), and right knee (RK) connect the upperand the lower extremities, among others.

The system can ask the sport enthusiast to squeeze a strength gauge,piezoelectric sensor, or force sensor to determine force applied duringsqueeze. The user holds the sensor or otherwise engages the sensor. Theuser then applies and holds a force (e.g., compression, torque, etc.) tothe sensor, which starts a timer clock and triggers a sampling startindicator to notify the user to continue to apply (maximum) force to thesensor. Strength measurements are then sampled periodically during thesampling period until the expiration of time. From the sampled strengthdata, certain strength measurement values are selected, such as themaximum value, average value(s), or values obtained during the samplingperiod. The user can test both hands at the same time, or alternativelyhe may test one hand at a time. A similar approach is used to sense legstrength, except that the user is asked to pushed down on a scale todetermine the foot force generated by the user.

In one embodiment, exercise motion data acquired by the accelerometer ormulti-axis force sensor is analyzed, as will be discussed below, inorder to determine the motion of each exercise stroke during theexercise session (i.e., horizontal vertical or circular). In anotherembodiment for detecting exercise motion using accelerometer, the firstminimum discovered during the scanning is noted as the first xmin andconsidered to be the start of the first brushstroke. The first maximum xvalue following the first minimum x value is located and construed to bethe middle of the first exercise stroke (where exercise motion changesfrom one direction to the other). The next xmin value indicates the endof the first brushstroke and the beginning of the next brushstroke. Thecomputer records the data for each brushstroke and continues on throughthe data to find the next brushstroke, recording each successive motionin memory. For the first brushstroke, the maximum and minimum values ofthe x coordinate (xmax and xmin) are determined. The Y-directionlengths, Ly1 and Ly2, between the data points just before and just aftereach of xmax and xmin (xmax+1, xmax−1, and Xmin+1, xmin−1) are thendetermined. The length Lx along the x axis, between xmax and xmin, isalso determined. Next, if Lx is less than 2 and either Ly1 or Ly2 isgreater than one, then the motion is construed to be vertical. If Ly1and Ly2 are both less than one, then the motion is construed to behorizontal. Otherwise, the motion is construed to be circular.

Data obtained from the gyroscope, if one is used, typically does notrequire a complex analysis. To determine which side of the mouth isbeing brushed at a particular time, the gyroscope data is scanned todetermine when the rotational orientation is greater than 180 degrees,indicating the left side, and when it is less than 180 degrees,indicating the right side. As explained above, top and bottom and gumbrushing information can also be obtained, without any calculations,simply by examining the data. The time sequence of data that is acquiredduring exercise and analyzed as discussed above can be used in a widevariety of ways.

In one embodiment, the accelerometers distinguish between lying down andeach upright position of sitting and standing based on the continuousoutput of the 3D accelerometer. The system can detect (a) extended timein a single position; (b) extended time sitting in a slouching posture(kyphosis) as opposed to sitting in an erect posture (lordosis); and (c)repetitive stressful movements, such as may be found on somemanufacturing lines, while typing for an extended period of time withoutproper wrist support, or while working all day at a weight liftingexercise, among others. In one alternative embodiment, angular positionsensors, one on each side of the hip joint, can be used to distinguishlying down, sitting, and standing positions. In another embodiment, thesystem repeatedly records position and/or posture data over time. In oneembodiment, magnetometers can be attached to a thigh and the torso toprovide absolute rotational position about an axis coincident withEarth's gravity vector (compass heading, or yaw). In another embodiment,the rotational position can be determined through the in-doorpositioning system as discussed above.

To improve a golf swing, the complex motion of the body first startswith the stance. The system checks that the golfer has a low center ofgravity to remain balanced throughout the swing path. The swing startswith the arms moving back in a straight line. When the club head reachesthe level of the hip, two things happen: there is a stern wrist cockthat acts as a hinge along with the left knee (for a right handedswing), building up its torque by moving into the same line as the bellybutton before the start of the upswing. As the swing continues to thetop of the backswing (again for right handed golf swing), the golfer'sleft arm should be perfectly straight and his right arm should be hingedat the elbow. The downswing begins with the hips and the lower bodyrather than the arms and upper body, with emphasis on the wrist cock. Asthe golfer's hips turn into the shot, the right elbow will drop straightdown, hugging the right side of the golfer's torso. As the right elbowdrops, the wrists begin to snap through from the wrist cock in thebackswing. A solid extension of the arms and good transfer of bodyshould put the golfer leaning up on his right toe, balanced, with thegolf club resting on the back of the golfers neck. Importantly, all ofthe movements occur with precise timing, while the head remainscompletely still with eyes focused on the ball throughout the entireswing.

The system can identify illnesses and prevent overexertion leading toillnesses such as a stroke. Depending on the severity of the stroke,sport enthusiasts can experience a loss of consciousness, cognitivedeficits, speech dysfunction, limb weakness, hemiplegia, vertigo,diplopia, lower cranial nerve dysfunction, gaze deviation, ataxia,hemianopia, and aphasia, among others. Four classic syndromes that arecharacteristically caused by lacunar-type stroke are: pure motorhemiparesis, pure sensory syndrome, ataxic hemiparesis syndrome, andclumsy-hand dysarthria syndrome. Sport enthusiasts with pure motorhemiparesis present with face, arm, and leg weakness. This conditionusually affects the extremities equally, but in some cases it affectsone extremity more than the other. The most common stroke location inaffected sport enthusiasts is the posterior limb of the internalcapsule, which carries the descending corticospinal and corticobulbarfibers. Other stroke locations include the pons, midbrain, and medulla.Pure sensory syndrome is characterized by hemibody sensory symptoms thatinvolve the face, arm, leg, and trunk. It is usually the result of aninfarct in the thalamus. Ataxic hemiparesis syndrome features acombination of cerebellar and motor symptoms on the same side of thebody. The leg is typically more affected than the arm. This syndrome canoccur as a result of a stroke in the pons, the internal capsule, or themidbrain, or in the anterior cerebral artery distribution. Sportenthusiasts with clumsy-hand dysarthria syndrome experience unilateralhand weakness and dysarthria. The dysarthria is often severe, whereasthe hand involvement is more subtle, and sport enthusiasts may describetheir hand movements as “awkward.” This syndrome is usually caused by aninfarct in the pons. Different patterns of signs can provide clues as toboth the location and the mechanism of a particular stroke. The systemcan detect symptoms suggestive of a brainstem stroke include vertigo,diplopia, bilateral abnormalities, lower cranial nerve dysfunction, gazedeviation (toward the side of weakness), and ataxia. Indications ofhigher cortical dysfunction—such as neglect, hemianopsia, aphasia, andgaze preference (opposite the side of weakness)—suggest hemisphericdysfunction with involvement of a superficial territory from anatherothrombotic or embolic occlusion of a mainstem vessel or peripheralbranch.

To detect muscle weakness or numbness, in one embodiment, the systemapplies a pattern recognizer such as a neural network or a Hidden MarkovModel (HMM) to analyze accelerometer output. In another embodiment,electromyography (EMG) is used to detect muscle weakness. In anotherembodiment, EMG and a pattern analyzer is used to detect muscleweakness. In yet another embodiment, a pattern analyzer analyzes bothaccelerometer and EMG data to determine muscle weakness. In a furtherembodiment, historical ambulatory information (time and place) is usedto further detect changes in muscle strength. In yet other embodiments,accelerometer data is used to confirm that the sport enthusiast is atrest so that EMG data can be accurately captured or to compensate formotion artifacts in the EMG data in accordance with a linear ornon-linear compensation table. In yet another embodiment, the EMG datais used to detect muscle fatigue and to generate a warning to the sportenthusiast to get to a resting place or a notification to a nurse orcaregiver to render timely assistance. The amplitude of the EMG signalis stochastic (random) in nature and can be reasonably represented by aGausian distribution function. The amplitude of the signal can rangefrom 0 to 10 mV (peak-to-peak) or 0 to 1.5 mV (rms). The usable energyof the signal is limited to the 0 to 500 Hz frequency range, with thedominant energy being in the 50-150 Hz range. Usable signals are thosewith energy above the electrical noise level. The dominant concern forthe ambient noise arises from the 60 Hz (or 50 Hz) radiation from powersources. The ambient noise signal may have an amplitude that is one tothree orders of magnitude greater than the EMG signal. There are twomain sources of motion artifact: one from the interface between thedetection surface of the electrode and the skin, the other from movementof the cable connecting the electrode to the amplifier. The electricalsignals of both noise sources have most of their energy in the frequencyrange from 0 to 20 Hz and can be reduced.

In one embodiment, the camera captures facial expression and a code suchas the Microsoft Emotion API takes a facial expression in an image as aninput, and returns the confidence across a set of emotions for each facein the image, as well as bounding box for the face, using the Face API.The emotions detected are anger, contempt, disgust, fear, happiness,neutral, sadness, and surprise. These emotions are understood to becross-culturally and universally communicated with particular facialexpressions. Alternatively, a marker for emotional arousal is galvanicskin response (GSR), also referred to as skin conductance (SC) orelectro-dermal activity (EDA). EDA modulates the amount of sweatsecretion from sweat glands. The amount of sweat glands varies acrossthe human body, being highest in hand and foot regions (200-600 sweatglands per cm2). While sweat secretion plays a major role forthermoregulation and sensory discrimination, changes in skin conductancein hand and foot regions are also triggered quite impressively byemotional stimulation: the higher the arousal, the higher the skinconductance. It is noteworthy to mention that both positive (“happy” or“joyful”) and negative (“threatening” or “saddening”) stimuli can resultin an increase in arousal—and in an increase in skin conductance. Skinconductance is not under conscious control. Instead, it is modulatedautonomously by sympathetic activity which drives human behavior,cognitive and emotional states on a subconscious level. Skin conductancetherefore offers direct insights into autonomous emotional regulation.It can be used as alternative to self-reflective test procedures,or—even better—as additional source of insight to validate verbalself-reports or interviews of a respondent. Based on the detectedemotion, the exercise can be increased, decreased, or stoppedaltogether.

Features of the auto-detection of exercise include the following:

An exercise system, including:

a processor running the motion analyzer and coupled to a wirelesstransceiver;

an accelerometer coupled to the processor; and

a kinematic motion analysis module executed by the processor to detectexercise type.

The system, including a plurality of smart modules mounted on anexerciser forming a mesh network.

The system where the electronic components, sensors, and interconnectsof the system monitor, record, process and/or transmit events ofinterest (such as accelerometers and gyroscopes for impact events,temperature sensors for temperature and/or temperature gradients,pressure sensors, moisture sensors, chemical sensors).

The system comprised for sensing and/or monitoring impact events wherethe sensors are accelerometers, gyroscopes, and/or pressure sensors.

The system comprised for sensing and/or monitoring and/or controllingongoing events where the sensors monitor temperature, temperaturegradients, motion, position, environmental or chemical levels, or othersuch information.

The system comprised for sensing events or other information includingmounting multiple distributed sensors for obtaining spatial and/ortemporal distribution in the data and/or multiple sensors sensingdifferent information and data.

The system including a camera and an image recognition module todetermine kinematic movement.

The system including a statistical recognizer to determine kinematicmovement.

The system, including a model-state that contains the extracted featuresof body signatures and other associated characteristics of bodysignatures.

The system including links connecting a root node (torso) with connectedjoint, left shoulder (LS), right shoulder (RS), left hip (LH), and righthip (RH), and left elbow (LE), right elbow (RE), left knee (LK), andright knee (RK) connect upper and lower extremities.

The system including a posture detection module.

The system, including a module to detect a lying down state and astanding state.

The system, including a hidden markov model module to detect musclemovement and exercise pattern.

The system including optimizing tennis shots to improve serve,groundstroke, volley, half volley, smash, forehand, backhand, flat, sidespin, block, slice, topspin shot, lob, passing shot, dropshot,cross-court shot, down-the-line shot.

The system, including an electromyography (EMG) sensor to detect musclestrength or weakness.

The system, including an emotion detector wherein an exercise can beincreased, decreased, or stopped based on detected emotion.

The system, wherein the detector comprises video detection of faces or aGSR sensor.

The system including a cloud storage to receive sensor data.

The system, including a golf training module that checks that a golferhas a low center of gravity to remain balanced throughout a swing path,that a swing starts with the arms moving back in a straight line, andwhen a club head reaches the level of the hip, a wrist cock acts as ahinge along with the left knee (for a right handed swing), building uptorque by moving into the same line as the belly button before the startof the upswing. As the swing continues to the top of the backswing(again for right handed golf swing), the golfer's left arm is straightand a right arm is hinged at the elbow.

The system, wherein the golf training module checks that a downswingbegins with the hips and the lower body and as the golfer's hips turninto the shot, the right elbow drops down, hugging the right side of thegolfer's torso and wrists begin to snap through from the wrist cock inthe backswing.

The system, including a soccer training module with kinematics of ballcontrol, dribbling, passing, crossing, shooting, heading, volleying,taking throw-ins, penalties, corner kicks and free kicks, tackling,marking, juggling, receiving, shielding, clearing, and goalkeeping.

The system, including a basketball training module with kinematics ofcrossover dribble, behind back, pull back dribble, low dribble, basicdribble, between legs dribble, Overhead Pass, Chest Pass, Push Pass,Baseball Pass, Off-the-Dribble Pass, Bounce Pass, Jump Shot, Dunk, Freethrow, Layup, Three-Point Shot, Hook Shot.

The system, including a baseball training module with kinematics ofHitting, Bunting, Base Running and Stealing, Sliding, Throwing, FieldingGround Balls, Fielding Fly Balls, Double Plays and Relays, Pitching andCatching, Changing Speeds, Holding Runners, Pitching and PitcherFielding Plays, Catching and Catcher Fielding Plays.

Data from multiple exercise sessions may be collected and used tocompile a history of the user's habits over an extended period of time,enabling the user's trainer to better understand user compliance issues.The trainer can review the data with the user and view the animations ofthe user's exercise sessions during an office visit, allowing thetrainer to better instruct the user in proper brushing technique. Thetrainer can also review the patient's brushing history over time, todetermine whether the patient's exercise technique is improving.

The sensor 14 can be integrated into objects already associated with thesporting activity. In one aspect, the sensing unit is integrated intothe ski boot or other boot. In another aspect, the sensing unit isintegrated into the binding for a ski boot or snowboarder boot. In stillanother aspect, the sensing unit is integrated into a ski, snowboard,mountain bike, windsurfer, windsurfer mast, roller blade boot,skate-board, kayak, or other sport vehicle. Collectively, the sportobjects such as the ski boot and the variety of sport vehicles aredenoted as “sport implements”. Accordingly, when the sensing unit is not“stand alone”, the housing which integrates the controller subsystemwith one or more sensors and battery can be made from the material ofthe associated sport implement, in whole or in part, such that thesensing unit becomes integral with the sport implement. The universalinterface is therefore not desired in this aspect.

One skilled in the art will appreciate that, for this and otherprocesses and methods disclosed herein, the functions performed in theprocesses and methods may be implemented in differing order.Furthermore, the outlined steps and operations are only provided asexamples, and some of the steps and operations may be optional, combinedinto fewer steps and operations, or expanded into additional steps andoperations without detracting from the essence of the disclosedembodiments.

The embodiments described herein may include the use of a specialpurpose or general-purpose computer including various computer hardwareor software modules, as discussed in greater detail below.

Embodiments described herein may be implemented using computer-readablemedia for carrying or having computer-executable instructions or datastructures stored thereon. Such computer-readable media may be anyavailable media that may be accessed by a general purpose or specialpurpose computer. By way of example, and not limitation, suchcomputer-readable media may include tangible computer-readable storagemedia including RAM, ROM, EEPROM, CD-ROM or other optical disk storage,magnetic disk storage or other magnetic storage devices, or any otherstorage medium which may be used to carry or store desired program codein the form of computer-executable instructions or data structures andwhich may be accessed by a general purpose or special purpose computer.Combinations of the above may also be included within the scope ofcomputer-readable media. Computer-executable instructions comprise, forexample, instructions and data which cause a general purpose computer,special purpose computer, or special purpose processing device toperform a certain function or group of functions. Although the subjectmatter has been described in language specific to structural featuresand/or methodological acts, it is to be understood that the subjectmatter defined in the appended claims is not necessarily limited to thespecific features or acts described above. Rather, the specific featuresand acts described above are disclosed as example forms of implementingthe claims. As used herein, the term “module” or “component” may referto software objects or routines that execute on the computing system.The different components, modules, engines, and services describedherein may be implemented as objects or processes that execute on thecomputing system (e.g., as separate threads). While the system andmethods described herein may be preferably implemented in software,implementations in hardware or a combination of software and hardwareare also possible and contemplated. In this description, a “computingentity” may be any computing system as previously defined herein, or anymodule or combination of modulates running on a computing system. Allexamples and conditional language recited herein are intended forpedagogical objects to aid the reader in understanding the invention andthe concepts contributed by the inventor to furthering the art, and areto be construed as being without limitation to such specifically recitedexamples and conditions. Although embodiments of the present inventionshave been described in detail, it should be understood that the variouschanges, substitutions, and alterations could be made hereto withoutdeparting from the spirit and scope of the invention.

What is claimed is:
 1. A voice assistant device comprising: at least onemicrophone to receive audio; a voice assistant processor to execute afirst security level authentication and based on predeterminedoperations, if encountering a secured request, execute a second securitylevel authentication by checking a voice signature of a user andchecking a camera view of a user, if the second security levelauthentication is unsuccessful, increase a counter to specify a periodto wait before allowing another authentication; and a secured systemremote from the voice assistant processor and in communication with theprocessor in the voice assistant, the secured system is controlled bythe secured voice request that passed the second level authentication.2. The device of claim 1, wherein a phone and a speech systemauthenticate a user before providing access to the secured system. 3.The device of claim 1, further comprising a smart phone to provide anadditional authentication or confirmation before granting supervisorlevel access to the device by using a voice command.
 4. The device ofclaim 1, comprising a mobile device with a public user key in anencrypted communication with the processor, wherein the processorauthenticates the user based on a private user key.
 5. The device ofclaim 1, wherein the device communicates with a remote server to analyzeuser dependent voice commands.
 6. A method comprising: receiving audio,via one or more microphones; processing a signal representation of theaudio, via a voice assistant processor, to execute a first securitylevel authentication, and if encountering a secured request, execute asecond security level authentication by checking voice signature of auser and authenticating the user by turning on a camera, and if thesecond security level authentication is unsuccessful, increase a counterto specify a period to wait before allowing another authentication; andif the secured request passed the second security level authentication,granting access to a secured system remote from the voice assistantprocessor.
 7. The method of claim 6, comprising processing the signalrepresentation to parse speech in the signal representation.
 8. Themethod of claim 6, further comprising processing the signalrepresentation to reduce double talk detected in the signalrepresentation, in conjunction with cancelling the acoustic echoesdetected in the signal representation.
 9. The method of claim 6, whereinprocessing the signal representation comprises substantially cancellingacoustic echoes detected in the signal representation, and then parsingspeech in the signal representation.
 10. A device comprising: at leastone microphone to receive audio; a voice assistant processor mounted toprocess a signal representation of the audio to (a) recognize speech inthe signal representation of the audio, (b) substantially cancelacoustic echoes detected in the signal representation of the audio, (c)execute a first security level authentication, (d) if encountering asecured request, execute a second security level authentication byauthenticating a user based on images from a camera, and (e) if thesecond security level authentication is unsuccessful, increase a counterto specify a period to wait before allowing another authentication; anda secured system remote from the voice assistant processor and incommunication with the processor in the voice assistant, the securedsystem being enabled or disabled by the secured request that passed thesecond level authentication.
 11. The device of claim 10, comprising agateway coupled to the processor to communicate with home appliances,the gateway communicating with a plurality of home area network (HAN)protocols the gateway translating HAN data from one protocol to anotherand providing interoperability among appliances, wherein the processoris adapted to communicate using the gateway with an appliance having thesecond security level authentication requiring additional check prior togranting full control of the device by using voice commands.
 12. Thedevice of claim 10, comprising a camera.
 13. The device of claim 10,further comprising a smart phone to provide an additional authenticationor confirmation before granting full control of the device by usingvoice commands.
 14. The device of claim 10, comprising a gatewaycommunicating with a plurality of home area network (HAN) protocolsincluding Zigbee (IEEE 802.15.4), Bluetooth (IEEE 802.15.1) and WiFi,the gateway translating Zigbee, Bluetooth and WiFi data from oneprotocol to another.
 15. The device of claim 10, comprising imagerecognition code.
 16. The device of claim 10, wherein at least onespeaker and the at least one microphone are coaxially aligned.
 17. Thedevice of claim 10, further comprising a sound distribution conearranged inside of a housing to distribute the sound emitted from atleast one speaker.
 18. The device of claim 17, wherein the at least onespeaker and the sound distribution cone are coaxially aligned.
 19. Thedevice of claim 17, wherein the sound distribution cone directs thesound at least partially in a radial outward direction.
 20. The deviceof claim 17, wherein the sound distribution cone directs the soundoutward from the housing proximal to a base.